Is it possible to use Conditional Reading for web-worker contexts? For example, I'm using prosemirror in one of my files. Unfortunately, prosemirror expects a global document variable, which is not available in web worker context. This makes it impossible to require the prosemirror namespace, because it throws an exception inside the webworker. I can definitely restructure the app into separate modules, but before doing that I just wanted to check because the Conditional Reading option would be much easier.

No worries, just solved it by patching the underlying prosemirror library.

@mjmeintjes no it only works per build. restructuring the modules is the way to go I think

https://twitter.com/DefnPodcast/status/991238980174798848

I am trying to see if we can port Calva to Clojurescript one file at a time. But I’m having troubles getting the require to work for requiring JS files. I have structured the src directory the same way as under *Requiring JS* in the User Guide: https://shadow-cljs.github.io/docs/UsersGuide.html#_requiring_js

require is very problematic. ES6 works much better if you can use that instead

Not sure I follow. It is (:require []) I don’t get to work.

oh. how so?

Trying to require a file utilities.js at the root of the src/js directory from a cljs file in a subdirectory to src/main. But whatever I type except for "utilities" fails, Shadow says no:

The required JS dependency “../utilities” is not available, it was required by “repl/message.cljs”.

hmm might be a bug in the resolver code

try ./../utilities?

Tried it, didn’t work. Even tried /../utilities 😃

ah hang on. the JS handling currently hides JS errors pretty badly

I asked shadow-cljs to be verbose, but it doesn’t give any more clues to this particular issue.

yeah hang on. looking into it

Hanging on. Many thanks!

which config are you using?

{:source-paths
 ["src/main"]

 :dependencies
 [[org.clojure/core.async  "0.4.474"]
  [cider/cider-nrepl "0.16.0"]
  [org.clojure/tools.nrepl "0.2.12"]]

 :builds
 {:calva
  {:target :node-library
   :output-dir "out-cljs"
   :output-to "src/js/main.js"
   :exports {:message repl.message/message}
   :modules {:main {:entries [repl.message]}}
   :devtools {}}}}

repl.message/message is a map with symbols from the module. I am hoping I can access that map from js code.

uhm did you reorganize your files?

I just forked master https://github.com/PEZ/clojure4vscode

are the JS files still in src but you added a src/main?

ah

https://github.com/PEZ/clojure4vscode/blob/wip-cljs-toolchain/shadow-cljs.edn

src/js must also be in :source-paths

but don't do :output-to "src/js/calva/main.js"

:output-to "dist/calva.js" or so and then use that as the main in package.json

Thanks. The problem with where to output-to is that main in package.json must remain ./src/js/extension for now.

that won't work though

So I must start with converting extension.js then maybe…

Now I get this:

[:calva] Build failure:
NullPointerException:
	shadow.build.classpath/find-js-resource (classpath.clj:1017)
	shadow.build.classpath/find-js-resource (classpath.clj:998)
	shadow.build.resolve/eval14123/fn--14126 (resolve.clj:192)
	clojure.lang.MultiFn.invoke (MultiFn.java:243)
	shadow.build.resolve/resolve-string-require (resolve.clj:227)
	shadow.build.resolve/resolve-string-require (resolve.clj:210)
	shadow.build.resolve/resolve-require (resolve.clj:393)
	shadow.build.resolve/resolve-require (resolve.clj:386)

I just got it to compile fine with

{:source-paths
 ["src/cljs"
  "src/js"]

 :dependencies
 [[org.clojure/core.async  "0.4.474"]
  [cider/cider-nrepl "0.16.0"]
  [org.clojure/tools.nrepl "0.2.12"]]

 :builds
 {:calva
  {:target :node-library
   :output-to "dist/calva.js"
   :exports {:message repl.message/message}
   :modules {:main {:entries [repl.message]}}
   :devtools {}}}}

how do I run this though?

Open the project in Code and hit F5.

That’s what I have too, except that I removed the :modules map. Will try with putting it back.

no don't it isn't used

only :exports is relevant

ok I got the NPE. tring to figure out what that is all about

ah hehe

doh

do you really want separate classpaths?

You could just use src/main or something and put everything in there

Probably not.

the problem is that the files are at the top-level

I would suggest making a src/main/calva folder and putting all the files in there

src/main/calva/utilities.js and so on

Trying it.

Do I need the main?

no but it is generally considered best practice in the JVM world

so you can add src/test src/dev and so on later

Makes sense

fixing the NPE now

yeah so the main problem you are going to run into the the use of require and module.exports

What should I use instead?

ES6 import/export

Thanks. Is this what is causing the NPE?

no the NPE is caused by the top-level file

so it is in /utilities and then tries to resolve ./state

it first tries to get the parent of /utilities which is nil

it expects there to be a folder 😛

So if I use source path src/main and the namespace calva.repl.message it might work?

That’s a nicer namespace anyway. 😃

so there are two parts to this problem

who is in charge of the build and who is in charge of loading the code?

it works best if shadow-cljs has control over both equally

the problem with your current approach is that shadow-cljs is building it but vscode is loading it differently

const { message } = require('../../calva/main');

this breaks

because the compile file is a completely separate universe basically

with ES6 you can do import X from "goog:calva.repl.message";

basically directly refer to a CLJS namespace

hmm don't know how to explain this in a simple example

That is what I wanted to do to begin with.

the goog: prefix is a special syntax from the closure compiler to access namespaces

I will look ES6 some more.

so I imagine you want to rewrite this one file at a time

Yes, that is the “plan”.

and JS + CLJS talking to each other

so the CLJS can import the JS and JS can use the CLJS

for this to work properly you need ES6 import/export

which should be a simple translation

Yeah, I should have been starting that translation long ago.

A minimal cljs main entry point which just uses the current extension.js should maybe solve the problem with where to output-to.

Anyway, if manualy I move the js file created by shadow to the root of where everything else lies (and use the main namespace calva to avoid the NPE) then I get things to compile and the extension to load. But I access the exported map incorrectly atm.

I now pushed an update to the wip-cljs-toolchain branch with these changes. If you want to try it, just move the generated dist/calva.js to src/main/calva before reloading the extension host.

@pez hang on I got it working. just running into a few bugs

It is fun seeing Calva activate in the project where it is being developed. =)

I’ll be away from the computer for some hours now. Hoping I’ll be home in time so I can continue this tonight.

@pez, @thheller did this https://github.com/thheller/clojure4vscode/tree/migrate-to-es6/src/main/calva It’s a little bit late here so once you get back I will be away

when initing a bootstrapped compiler env with (boot/init state {.... is it posible to pass options like in a regular build def

...
:my-build {...
:compiler-options {:my-flag true}
} ....
} 

@fj.abanses you are passing the state to init

just set in it state?

(defonce c-state (cljs/empty-state))
(swap! c-state assoc-in [:options :my-option ] false)

(defonce _ (boot/init c-state
                    {:path         "/js/bootstrap"
                     :load-on-init '#{b2repl.user}}
                    (fn []))

it is. boot/init is just about loading pre-compiled code. does nothing beyond that

thanks @thheller 👍

Getting this with shadow-cljs check:

Failed reading cache for scratchpad.core: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array (start marker at [Source: [email protected]; line: 1, column: 1028532])
 at [Source: [email protected]; line: 1, column: 1033075]
Failed writing cache for scratchpad.core: java.lang.Exception: Not supported: class java.util.regex.Pattern

And shadow-cljs check takes quite a while to finish, which may as well just be the Closure compiler, but if it's slow due to the cache failing for some reason, it'd be great if it could be sped up.

compile doesn't complain about cache?

or release? or watch?

watch doesn't, release dunno, checking now with newest version

is this open source code? wonder where the pattern is coming from? probably somewhere in meta?

easy to fix. just wondering why it only occurs in check

okay, watch isn't complaining, release is:

Failed writing cache for scratchpad.core: java.lang.Exception: Not supported: class java.util.regex.Pattern

It's not open source yet, will be soon

stop the watch and do a single compile please

just want to confirm that all phases complain. doesn't make sense for watch not to complain

I tested release first, before starting the watch, will do again, hang on...

watch just uses the cache once otherwise it'll be in memory already

so it won't complain about cache failing

Failed writing cache for scratchpad.core: java.lang.Exception: Not supported: class java.util.regex.Pattern

On shadow-cljs compile

Could it be relevant that 1. this a .cljc file with macros and reader conditionals and 2. I'm using the lein integration with shadow?

I'm also getting these:

WARNING: failed to inspect resource "jar:file:/[....]/scratchpad/core.cljc", it will not be available.

No idea if it's related in any way, but it's the same file.

maybe a bad ns form?

check the logs

if it doesn't contain any CLJS code just make it a .clj file

wait

it fails compilation despite being not available?

that doesn't make sense

Compilation succeeds actually

so you didn't get the inspect resource error?

I'm confused

It's just a WARNING, the inspect resource thing

It then goes on to display a build completed message

hold on for a second

IF! you get WARNING: failed to inspect resource "jar:file:/[....]/scratchpad/core.cljc", it will not be available. the file will not be available

and thus NEVER can be compiled successfully

so you either got that error some time ago and fixed it in the meantime

or something else is going on

`

❯ shadow-cljs compile scratchpad                                                                                                                      1s 862ms
shadow-cljs - config: /Users/gnl/Code/re-frame-playground/shadow-cljs.edn version: 2.3.14
shadow-cljs - running: lein with-profile +shadow run -m shadow.cljs.devtools.cli --npm compile scratchpad
May 01, 2018 5:00:53 PM clojure.tools.logging$eval870$fn__874 invoke
WARNING: failed to inspect resource "jar:file:/Users/gnl/.m2/repository/gnl/scratchpad/0.1.0-SNAPSHOT/scratchpad.1.0-SNAPSHOT.jar!/scratchpad/core.cljc", it will not be available.
May 01, 2018 5:00:53 PM clojure.tools.logging$eval870$fn__874 invoke
WARNING: failed to inspect resource "jar:file:/Users/gnl/.m2/repository/zprint/zprint/0.4.7/zprint-0.4.7.jar!/zprint/main.cljc", it will not be available.
[:scratchpad] Compiling ...
May 01, 2018 5:01:00 PM com.google.javascript.jscomp.LoggerErrorManager println
WARNING: node_modules/highlight.js/lib/languages/tp.js:14: WARNING - String continuations are not recommended. See 
    begin: '(AR|P|PAYLOAD|PR|R|SR|RSR|LBL|VR|UALM|MESSAGE|UTOOL|UFRAME|TIMER|\
           ^

May 01, 2018 5:01:00 PM com.google.javascript.jscomp.LoggerErrorManager printSummary
WARNING: 0 error(s), 1 warning(s)
------ WARNING #1 --------------------------------------------------------------
 File: node_modules/highlight.js/lib/languages/tp.js:14:11

 String continuations are not recommended. See 
--------------------------------------------------------------------------------

nil
Failed writing cache for scratchpad.core: java.lang.Exception: Not supported: class java.util.regex.Pattern
[:scratchpad] Build completed. (602 files, 483 compiled, 0 warnings, 49.68s)

I assume you are working on scratchpad so why is scratchpad.1.0-SNAPSHOT.jar on the classpath?

the SNAPSHOT jar probably contains an older version which is then overwritten by your source files

so everything is fine

still confusing but fine

I had the jar installed with lein install because I was messing around with lein checkouts before shadow-cljs and now I'm directly referring the source-path of the current version

but why is it on the classpath?

ah ok you have it in your deps?

ah nvm I get it now 😛

Yes, lein checkouts supposedly needs that

I'm not using nvm. 🙂

nevermind 😉

What a mess. 🙂

ok the jar is just outdated but not a problem since you have the sources directly available

that is fine

Okay, so the .jar is irrelevant in this case and I can safely ignore the message?

just ignore the warning then

Okay.

confusing as hell if you delete a file but still leave it in the jar

but ok

I'll remove the .jar

No need for lein checkouts anymore

spent an hour or so once debugging this. couldn't figure out how it could still load a file that I just deleted 😛

anyways ... the cache issue

Those are the best issues, when you start questioning the very fabric of reality. 🙂

Probably best to open an issue when the source is online

Before we waste time guessing what might be going on

just grep the source for #" please

is it used in any metadata or so?

just need to find out where the regex is coming from

seems weird that it would leak into the compiler data but could be totally fine

It's used in a spec

and in a (comment) block

I can try removing the (comment) block

that shouldn't do anything

it is never compiled

I know how to fix it but I want to find out WHY it happens first

I'll remove it from the spec too, hang on

if must be a regex in that file so it can't be in too many places

It's just those two, comment and spec

(s/def ::foo #"hello world") this is fine so thats not it

checking...

Hah! Got it:

(s/def ::color (s/or :keyword keyword?
                     :literal (s/and string?
                                     #(re-matches #"#[a-fA-F0-9]+" %)
                                     #(or (= (count %) 7) (= (count %) 4)))))

indeed! thx

wonder why that ends up in the analyzer data

That keyword ends up in some metadata and you mentioned something about metadata before, is that a good direction?

ah right. its the specs getting serialized. forgot about that.

The caching working doesn't make any difference to how long shadow-cljs check takes though, so apparently that's just the Closure compiler needing a while. Is there any way to speed this up, do you think? Somehow just recheck changed files or something or does it need to go over the whole project every time?

define long?

31s

wow hmm shouldn't take that long

take a second or two in my code

To describe the environment - it's a playground project with many dependencies one of which is the library that I'm currently working on.

Based on re-frame

and lots of the warnings are in re-frame and reagent

[:scratchpad] Build completed. (602 files, 483 compiled, 0 warnings, 49.68s)

is that all CLJS?

or lots of JS deps as well?

49sec seems really slow as well

Just cljs

but yes check always checks all files. can't do a partial check.

Which makes sense in a way, but some sort of caching would be nice, I'd love to be able to do the type-checking at a similar speed as hot-reloading.

that would take a pretty significant rewrite of the closure compiler

its not currently set-up to do this live

you could make an extra build

that only includes the namespace you want to check

so not your main entry for everything

only a partial subset

ie. shadow-cljs check-ns cljs.core or something like that

not sure how useful that would be in your case

but the way the closure compiler does this means that you must provide all the sources and dependencies

and currently you are only allowed to run it once. second attempt will fail.

so you must throw away the previous instance and start fresh

ie. no cache (for GCC)

cljs is cached regardless

maybe they support incremental checks and I just haven't found it in the API yet

not sure

just published [email protected]. should fix the cache issue

Okay, this clears up a few things, thanks. I don't think a single-ns check will do the trick for my use case, but I'll let you know when I finish my thing, it'll be clearer what I have in mind and then we can brainstorm some more about how best to approach this if at all possible.

Alright, testing...

probably won't make GCC any faster

how long does release take?

50+ seconds

Let me check again...

there's a great npm library https://github.com/gtanner/qrcode-terminal

Okay, so:

shadow-cljs check
[:scratchpad] Build completed. (161 files, 0 compiled, 0 warnings, 4.80s)
34s 786ms

shadow-cljs release
[:scratchpad] Build completed. (161 files, 130 compiled, 0 warnings, 41.58s)
55s 530m

What's the discrepancy between the time reported in build completed and the actual time it took?

I'm assuming in the case of check that's where the actual checking happens

btw the caching issue's fixed in 2.3.15, thank you!

try running with --verbose

hmm

uhm wait

do you not have a server running?

I do when watching, haven't really done any release builds yet and just started checking

definitely always have a separate npx shadow-cljs server running

makes all commands a whole lot faster since you don't have to pay for the startup each time

probably takes 20sec to just start things and warm up before anything actually related to check happens

Oh wow, clearly I've severly underestimated the startup time of the whole thing. (shadow/check ...) in the REPL is done in 14seconds now.

Still not great, but way better.

yeah startup is painfully slow

Thank you!

14sec is still not ideal. try (shadow/check :scratchpad {:verbose true})

most of the time is probably spent in check itself

probably can't make that faster

Yup, will probably have to cut down on dependencies, but will check verbose true first

Yeah, about 12 seconds on checking, the rest on loading stuff from the cache

hmm yeah. dunno if there are flags in the closure compiler I'm supposed to be setting

checksOnly supposedly turns off all the slow stuff

but maybe not everything

its not super reliable to begin with anyways

Maybe :closure-warnings makes a difference in that it doesn't only control reporting but actual checking?

in most cases it only controls whether a warning is emitted or not

sometimes it does enable the entire check too though

gotta go. I'll be back in a few hours.

Cool, thank you for the help!

Thanks for the help with migrating Calva imports/exports to ES6, @thheller! I have merged it into my wip branch now and it compiles and the extension loads. I pushed my branch with the changes. The extension doesn’t quite work yet, commands die silently. I guess the message module looked resonably simple to start the porting with, but it is also very central to most everything Calva does. 😃

But I’m on it and hope to be able to merge this into master soon.

Hmm, I have made some little progress, but it seems that utilities.js is sometimes not imported correctly, I get errors saying things like module$calva$utilities.X is not a function from places like hover.js and messages.cljs. However shadow.js seems to get a correct import, because it can use utilities.js functions.

And now I am too tired to think so this will have to rest a while. 😃

@doglooksgood sorry about that. fixed in 2.3.16

@pez yeah the cjs-to-es6 tool I used to migrate the exports did some weird stuff

definitely not correct it those and just accidentally works here and there

So you think I can get it working by fixing the imports manually? They look right to me. (I don’t see any difference between the imports that works and those that don’t.)

problem is that it used export default { some, stuff, ... }

this works if the other file does import x from "./utilities" and then does x.some

doesn't work if it does import { some, stuff } from "..."

can either use export without the default and adjust the others

to use import * as util from "./utilities" and then util.<thing>

for some reason ES6 doesn't allow "partial" refers. either all or nothing. no idea why.

Yes, now I got some more stuff to work. Thanks for the pointers!

There. Now it seems most stuff works. Maybe all stuff. I am really too tired to figure that out right now. 😃 Anyway, pushed the changes to the wip branch for anyone who cares. Calva is on it’s way to be ported to Clojurescript, thanks to shadow-cljs!

I had to do a funny trick. The new message.cljs module of course produces proper clojure maps. But the bencoder(.js) module of course has none of that. So I made a small migrate.cljs module that converts it back to a Javascript object. I use it from client.js before giving the message to bencoder. If I can port client.js to Clojurescript I will be able to remove this migration module, I think.

Hi. Every now and again I get the following error printed by shadow-cljs:

[2018-05-01 23:27:27 - SEVERE] Unhandled REPL handler exception processing message {:ns user, :op eval, :code (ns learning.web.web-worker.core
  (:require [learning.web.module.core :as m]
            [keechma.toolbox.pipeline.core :as pp]
            [keechma.controller :as controller]
            [learning.js-utils :as j]
            [keechma.toolbox.pipeline.controller :as pp-controller]
            [goog.object :as go]))
, :file /home/matthys/projects/personal/learning/src/learning/web/web_worker/core.cljs, :line 1, :column 1, :session 6e5b0609-56d9-4d5f-9b08-91f627139e22, :id 15}
java.lang.NoClassDefFoundError: Could not initialize class shadow.cljs.devtools.errors$user_friendly_error$fn__16179
        at shadow.cljs.devtools.errors$user_friendly_error.invokeStatic(errors.clj:290)
        at shadow.cljs.devtools.errors$user_friendly_error.invoke(errors.clj:289)