Fork me on GitHub
#cljs-dev
<
2018-03-18
>
john01:03:41

I don't see that aot-cache-core is ever called outside of script/aot

john01:03:13

But I only ever see :aot-cache false for coming through cached-core for cljs.core.cljs

john02:03:55

Ah, it was turned off 🙂

mfikes02:03:28

I’m beginning to wonder if https://dev.clojure.org/jira/browse/CLJS-2646 is a valid ticket.

mfikes02:03:42

I’ve closed https://dev.clojure.org/jira/browse/CLJS-2646 with a note as to why I concluded I was wrong when I wrote it. I’m still curious about https://dev.clojure.org/jira/browse/CLJS-2671 which is likely unrelated.

mfikes04:03:51

^ Have a solution for CLJS-2671; assembling up a patch for consideration

mfikes12:03:56

Wow, since this ^ change eliminates double compilation of JAR code for a completely fresh build (including no ~/.cljs), it drops my initial build from 35.6 s to 28.7 s for a React Native project that uses Om and other JAR deps.

r0man14:03:25

Tested 1.10.191 with a project that uses code splitting and cljs.loader without success. Running into CLJS-2650 again. https://dev.clojure.org/jira/browse/CLJS-2650

r0man14:03:47

It looks like the loaded.cljs doesn't get moved from the JAR to disk because requires-compilation? returns false here: https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/closure.clj#L593

dnolen14:03:38

@r0man aot-cache is false by default so how do you get there?

r0man14:03:28

hi david, to which line?

dnolen14:03:12

Where you’re pointing doesn’t make sense or do you mean requires-compilation is wrong generally?

r0man14:03:35

the compile-file at the end of this function alwas gets evaluated

mfikes14:03:07

Yeah, @r0man’s line is off a little

mfikes14:03:26

I can repro, and get passed that by hacking the call to look like

(comp/requires-compilation? jar-file out-file (update opts :cache-key (fn [x] (or x :bogus))))

mfikes14:03:00

But then it can't find bar.core in the module graph

r0man14:03:55

what I'm trying at the moment is to copy the cljs.loader from jar to disk just before the last compile-file in compile-from-jar.

mfikes14:03:21

The :bogus hack above does that, and gets past that problem.

r0man14:03:34

so compile-from-jar returns whatever compile-file returns, but skips the cache. but not sure if that is even right?

mfikes14:03:25

(If you look at compile-from-jar in 1.9.946 and now, you can see that the conditions have changed a little, but the :cache-key :bogus hack makes it behave more like 1.9.946

dnolen14:03:25

Whatever the problem is I don’t see how compile-from-jar can be the cause

r0man14:03:45

@dnolen Isn't compile-from-jar responsible for moving files from jar to disk? this also seems only to happen when you actually use clojurescript from a JAR.

dnolen14:03:16

-compile should take care of it

mfikes14:03:24

My hunch ws that, without the :bogus hack, jar-file-to-disk never gets called, and it jumps the gun, calling compile-file

dnolen14:03:30

So explanation required why it does not

r0man14:03:45

@mfikes I'm not sure I understand your :bogus hack 🙂

mfikes14:03:01

requires-compilation? has hard-coded logic to return false for this namespace unless you use the hack

dnolen14:03:11

Yes because we skip it

dnolen14:03:15

It must come last

dnolen14:03:22

See compile-loader

mfikes14:03:43

Yep, I'm just comparing the old and new behavior. Perhaps it was compiling it prematurely in 1.9.946? Dunno.

dnolen14:03:19

Nothing changed here except one thing

dnolen14:03:33

Before it was -compile or compile-file

r0man14:03:35

@dnolen yes, requires-compileation? retuns false in compile-from-jar. then the code to copy it from jar to disk doesn't get evaluated. but the last line ,`compile-file`, in compile-from-jar fails.

dnolen14:03:03

Now it is -compile plus compile-file

mfikes14:03:35

But the scope of the when cause it to be just compile-file

dnolen14:03:38

Oh I know the problem

dnolen14:03:54

We didn’t use requires-compilation? before

dnolen14:03:01

It was a variant

mfikes14:03:03

Right... that logic changed subtly

r0man14:03:48

I'm testing this at the moment

r0man14:03:25

The special case for cljs.loader in requires-compilation? is to prevent cljs.loader from beeing cached, right?

mfikes14:03:39

It is to delay it to the end, right?

mfikes14:03:47

The :cache-key :bogus hack is not the right solution, but if useful, here is what happens with it in place: https://gist.github.com/mfikes/969939e06e830bcd2504fb58aa285101

dnolen14:03:52

I think the right answer is to just skip it I’m surprised compile-file sees this

dnolen14:03:12

It also uses requires-compilation?

mfikes14:03:19

Ahh, figured it out. bar/core.clj should be bar/core.cljs

dnolen14:03:24

Also we need a test

mfikes14:03:34

I got it running with the hack 😝

r0man14:03:32

@mfikes ah the guide is wrong. I run in to this previously as well, but thought I made a typo

mfikes14:03:44

touch src/bar/core.clj right

mfikes15:03:28

(Even though I said, I got it running with the hack. I don't want to complain it is really successful. It brings up the browser, and it has a "Load Bar!" button, but I am unfamiliar with what is really supposed to happen at runtime at that point.)

r0man15:03:37

@mfikes you should see something in the console

dnolen15:03:50

My suggestion is first just remove the special case from requires-compilation?

dnolen15:03:17

Let it compile, it will be compiled again anyway

r0man15:03:33

ok, I'll try that

r0man15:03:28

any ideas for testing this? the current tests won't catch this, because loader.cljs is on the classpath/src path I think?

r0man15:03:49

do we already have tests that use the cljs.jar for something?

mfikes15:03:10

Yes, the cljs.main test path uses cljs.jar

mfikes15:03:33

(`script/test-cli`)

r0man15:03:56

ok, then I may just add the example from the code split guide as the test, ok?

mfikes15:03:23

@r0man script/test-cli really doesn't have a way to test browser interaction...

mfikes15:03:31

In fact, the browser won't start in CI

r0man15:03:32

@mfikes ok, I thought about just checking the output files. and that it "jsut" compiles. 🙂

mfikes15:03:03

Yeah, there is a way to check output files in the script/test-cli setup and make it fail if some arbitrary post-condition doesn't hold. A trick will be that it will block waiting for the browser to connect...

mfikes15:03:45

@r0man Perhaps if you just use -c and don't involve a -r in the test?

dnolen15:03:15

Yes I would keep it simple

r0man15:03:00

ok, I'll take a look

mfikes15:03:20

@r0man One idea to consider: The cljs-cli.test namespace is really testing cljs.main's core functionality. But it seems that the ability to run some end-to-end tests using script/test-cli might be useful going forward. Thinking that perhaps for this one, there could be a code-splitting.test namespace or somesuch that has a test or tests for this feature, and cljs-cli.test-runner could be revised to require and run that namespace's tests as well.

mfikes15:03:43

(I'm just thinking that we might find the tests being run here will grow over time, as it is an easy way to exercise the system in an end-to-end way, and if that happens, perhaps separation of test namespaces might be useful.)

mfikes15:03:48

Also, @r0man if you look at the way the tests are run, the REPL environment is specified as an optional command line argument when the tests are run. Not sure if that framework needs to be revised to accommodate skipping certain tests if they don't apply to a given target or REPL environment. (So far, all tests are applicable to all environments.)

mfikes15:03:28

Maybe code-splitting works in Node?

r0man15:03:00

never tried code splitting with node. I'll try it out ...

r0man15:03:52

@dnolen @mfikes I just removed the cljs.loader check in requires-compilation? and it seems to work with my project as well.

r0man15:03:10

now, I tried to test this with :aot-cache. I'm using :aot-cache true in my compiler options but ~/.cljs stays empty. did the direcory change?

r0man15:03:47

ok, now I see the cache. must have messed something up

mfikes15:03:09

:closure-warnings refresh now in the comprehensive 1.10.x doc list: https://gist.github.com/mfikes/bdbe214f03abac88ae384adb1ac26490

r0man17:03:52

@dnolen @mfikes I added a patch to https://dev.clojure.org/jira/browse/CLJS-2650 It uses cljs.main to compile the code splitting guide in :none and :advanced. At the moment this test is in cljs-cli.test. Should I move it somewhere else?

mfikes18:03:22

@r0man I think it is fine... if we find that this namespace is getting crowded, we can spit it apart later.

mfikes18:03:41

@r0man Your patch tests the ability to pass two --compile main opts simultaneously? I wasn't aware you can do that.

mfikes18:03:23

If I had to guess, the last one wins. (This is what happens if you pass two or more --re options.

r0man18:03:03

@mfikes That's the first time I'm using cljs.main. I tried it with only foo.core or only bar.core but I think only one of them ended up in the module info of cljs.loader.

r0man18:03:07

Let me check again ...

mfikes18:03:32

Interesting. Perhaps you've discovered a feature. 🙂

r0man18:03:40

No, I think you are probably right. I tried a couple of things to get this working and ended here. I'll remove the second one.

r0man18:03:59

Just checked, cljs.loader.module_infos look right with only one option.

r0man18:03:21

ok, patch updated

Garrett Hopper20:03:30

@dnolen So while trying to get a minimal test using only cljs.jar I realized that even the included test (https://github.com/clojure/clojurescript/blob/78b2395960767ea44b68ddd632d1dfe9a4957853/src/test/cljs_build/code-split/repl.clj) doesn't run with the latest versions. (Anything after r1.9.946 from what I can tell.)

Caused by: java.io.FileNotFoundException: The file out/cljs/loader.cljs does not exist.
        at cljs.compiler$compile_file$fn__4145.invoke(compiler.cljc:1558)
        at cljs.compiler$compile_file.invokeStatic(compiler.cljc:1523)
        at cljs.closure$compile_file.invokeStatic(closure.clj:556)
        at cljs.closure$compile_from_jar.invokeStatic(closure.clj:626)
        at cljs.closure$fn__5648.invokeStatic(closure.clj:647)
        at cljs.closure$fn__5648.invoke(closure.clj:632)
        at cljs.closure$fn__5571$G__5564__5578.invoke(closure.clj:511)
        at cljs.closure$compile_sources$iter__5758__5762$fn__5763.invoke(closure.clj:983)
        at clojure.lang.LazySeq.sval(LazySeq.java:40)
        at clojure.lang.LazySeq.seq(LazySeq.java:49)
        at clojure.lang.Cons.next(Cons.java:39)
        at clojure.lang.RT.next(RT.java:706)
        at clojure.core$next__5186.invokeStatic(core.clj:64)
        at clojure.core$dorun.invokeStatic(core.clj:3134)
        at clojure.core$doall.invokeStatic(core.clj:3140)
        at cljs.closure$compile_sources.invokeStatic(closure.clj:980)
        at cljs.closure$build.invokeStatic(closure.clj:2776)
        at cljs.build.api$build.invokeStatic(api.clj:204)
        at cljs.build.api$build.invoke(api.clj:189)
        at cljs.build.api$build.invokeStatic(api.clj:192)
        at cljs.build.api$build.invoke(api.clj:189)
        at user$eval38.invokeStatic(repl.clj:25)
        at user$eval38.invoke(repl.clj:25)
        at clojure.lang.Compiler.eval(Compiler.java:7062)
        at clojure.lang.Compiler.load(Compiler.java:7514)
        ... 9 more

mfikes20:03:41

@ghopper Not sure if you notice the above. Fixed on master.

Garrett Hopper20:03:13

@mfikes Sorry, I didn't see that. Though I did just now compile an uberjar from master. I'll try again—making sure I definitely have the latest.

mfikes20:03:25

Was only fixed in the last several hours.

Garrett Hopper20:03:24

@mfikes Works like a charm. I guess I didn't actually have the latest master. Thanks! 🙂

mfikes20:03:41

Yeah, it was only just discussed and resolve this morning (Eastern time US)

Garrett Hopper20:03:11

Great timing 🙂

Garrett Hopper20:03:05

Has anyone had issues with goog.module.ModuleLoader being undefined in goog.module.ModuleLoader.prototype.evaluateCode_? It's unable to dispatch an EvaluateCodeEvent event, because ModuleLoader is undefined.

Garrett Hopper20:03:05

I think it's because it's loading :cljs-base which is redefining goog.module. Is it normal for cljs.loader/load to also load :cljs-base?

Garrett Hopper20:03:07

While loading, there's an error on goog.globalEval for :cljs-base: Error: Namespace "goog.string" already declared. at Object.goog.provide (eval at goog.globalEval ( So why is the loader trying to load :cljs-base if that is already loaded by the script tag?

mfikes21:03:12

I'm really surprised that this seems to work. It makes me wonder if ClojureScript's persistent map avoids mutating itself at all when a revision based on it is made. Perhaps it does, and this code is skating on thin ice. Anyway, it bodes well if multthreading ever becomes a thing in JavaScript (in that ClojureScript might be within reach of supporting such a thing). https://gist.github.com/mfikes/9ef1908fff34c52a624f3dcf408d545f

john21:03:23

that's pretty cool

john21:03:54

That completely mixes java and javascript together

mfikes21:03:31

Right. Curiously powerful. Will be freakishly powerful if project Detroit happens.

john21:03:41

You could mix aribtrary clojure and clojurescript code together, I guess, in the same context?

john21:03:48

so weird

john21:03:50

But does this make all nashorn threads reentrant? So js's run to completion semantics go away?

john21:03:30

no more guarantees of safety

mfikes21:03:32

In addition to Rhino / Nashorn, this JavaScriptCore experiment is interesting https://webkit.org/blog/7846/concurrent-javascript-it-can-work/

mfikes21:03:04

Weren't you mentioning that in Baltimore?

john21:03:07

yeah I saw that. I like it but it sounds like a "big idea"

john21:03:29

Yeah, was working on the SharedArrayBuffer stuff, which is a similar vein

john21:03:45

but meltdown melted down our SharedArrayBuffer dreams

john21:03:28

Not that I was going to be able to build a slab allocator / garbage collector over SAB 🙂

john21:03:45

I did figure out a way to make a sort of "synthetic" future in cljs though

john21:03:45

you use a service worker. intercept synchronous xhr calls and hang the client webworker until some result is finished, then pass the result back in the response

john21:03:09

Then you can build futures where you can do things like (await (anything-asynchronous blah

john21:03:19

but wait for real 🙂

john21:03:14

The wait is not reentrant though... run to completion for the worker context still holds

john21:03:30

built a working version of it, but it's a little too complex

john21:03:03

and it's pretty egregious to require a service worker just to get a future

john22:03:25

interestingly, in a cluster of web workers, I think Clojure's three main concurrency primitives can be mostly implemented in cljs

john22:03:03

When SAB comes back, for sure

john23:03:40

and with run to completion semantics, you can be sure that cljs threads can't be interrupted, so you get some concurrency protections that Clojure provides for free. Makes implementation a bit easier I believe

john23:03:59

Also interesting, one could make the wesocket and webworker postMessage interface line up fairly squarely and have local and remote workers look the same to each other, allowing for networks of distributed cljs worker clusters pushing data across a WebRTC mesh

john23:03:20

just sayin 😉