@dnolen does compiler produce source map when compiling with modules? Here it says setting :source-map true should produce individual source maps for each module https://github.com/clojure/clojurescript/wiki/Compiler-Options#modules But when I set :source-map true I immediately get the following error:

Compiling "/projects/test/target/cljsbuild-main.js" failed.
java.io.FileNotFoundException: /projects/test/resources/public/js/compiled/out-min/cljs/core/constants.js (No such file or directory)

^ trying this from master

I’m guessing some weird interaction between modules and :optimize-constants

@roman01la can you try :source-map true :optimize-constants false?

@anmonteiro Will try tomorrow, going to sleep now. Thanks

let me try it locally

^ turns out this is not only a problem for :modules, but for the combination of :source-map and advanced compilation. Fixed in https://dev.clojure.org/jira/browse/CLJS-2169

^ the fix is quite simple. The bug was there all along, but manifested itself because source-on-disk was moved up

we were giving the constants JavaScriptFile a source-url when there really isn’t one so it would pass this check: https://github.com/clojure/clojurescript/blob/e86c36bb5408a0778bde6b56a292819d6d2af4c6/src/main/clojure/cljs/closure.clj?utf8=%E2%9C%93#L1701

oh hrm @juhoteperi was way ahead of me wrt. goog.module https://dev.clojure.org/jira/browse/CLJS-1543

@anmonteiro IIRC goog.module is transpiled the same way ES6 would by using the embedded goog/transpile.js at runtime

so probably not something we should ever emit

@anmonteiro I think that issue is about more supporting Closure libs that use goog.module, than changing what Cljs emits

I just had a build fail with 10:50:04 Caused by: java.lang.ClassCastException: clojure.lang.Var$Unbound cannot be cast to java.util.concurrent.Future

the stacktrace points at this line as the culprit https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/analyzer.cljc#L3697

failure seems to be intermittent as subsequent build at same rev succeeded

is this a known issue? if not I'll log a ticket

this is the full stacktrace http://sprunge.us/FXMT

@bronsa yeah log a ticket

:thumbsup:

@juhoteperi left feedback, option 3 seems sound to me.

I also forgot that the transform fn takes compiler opts - I think this is fine for sharing whatever needs to be shared

@dnolen But transform fns can't update the opts?

Should they be able to?

@juhoteperi I just mean shared info, if they need to share state, they can write to the compiler state

I suppose the other way is for transform fns to return a map with :ijs :opts

(need better names)

it doesn’t really protect anyone from clobbering any more than just writing to compiler state though

opened https://dev.clojure.org/jira/browse/CLJS-2171, unfortunately I don't have a minimal repro for this (we've only hit this issue twice in the last ~10 days, over many builds)

@bronsa and no parallel-build involved here?

parallel build is enabled

I've pasted the compiler opts we're using in the ticket

I doubt there is much need for js-transforms to be able to modify the opts anyway. Probably reading ijs/opts is enough and using compiler-state or just a atom is enough for state.

this may again be what I suggested which is that we dont lock a require of cljs.spec.alpha \cc @anmonteiro

@juhoteperi yeah I’m inclined to agree

Okay, I think the specs for the implementation are quite clear to me now. I'll write version to use symbols + functions today, and I'll leave the multimethod around for backwards compatibility.

great!

@bronsa I don’t know if it’s easy for you to test master in your setup but I locked require of cljs.spec.alpha in master

testing master is not the hard part, it's getting the bug to happen again unfortunately

i'll give master a spin anyway

^ The Lumo parallel build failure involved cljs.spec.gen.alpha/lazy-prims, so maybe this is the the same

@dnolen I just added a comment to https://dev.clojure.org/jira/browse/CLJS-2168. I'm not sure if there will be anything more to discuss about it, but feel free to hit me up here if it'll save you a minute. I'll be in and out all day.

@dnolen rebased https://dev.clojure.org/jira/browse/CLJS-2037

also, I’ll write something up about string-based requires to add to @mfikes blog post snippet for the next release

I’m actually thinking the next release will need at least 3 posts

1) aget/aset 2) :modules 3) :npm-deps + string based require

order matters less than, I think each of these deserve their own post

and the release post states that these are coming

and maybe release one same day

perhaps we can roll out posts before the actual release

that’s another option

to announce what’s coming

and I’m OK with that!

I’ll put something together for npm-deps + string requires

Here's the aget / aset draft. Knowing that it would be standalone, it could afford more content if there was some aspect we wanted to additionally mention, emphasize, etc. https://gist.github.com/mfikes/ba5e999eeef8295ee9d8d53af94ebc18

@mfikes can we please move that to a ClojureScript PR?

Yes

er clojurescript-site PR

It does not currently mention unsafe-get or detail goog.object/getValueByKeys, fir example. I'll put it in a PR where it is easier to review

On the previous post I had put the author as ClojureScript Team. We could decide for these posts whether that is continued, or if we put individual names on posts.

I think individual names is nicer to call out contributors

I think release notes ClojureScript team makes sense

@dnolen btw in the duplicate alias patch I don't check across require / require-macros on purpose. This is because of a cljs.core.async thing we noticed worked last year

yes I noticed that

Gotcha

Here is the aget / aset post PR: https://github.com/clojure/clojurescript-site/pull/101

thanks

Perforce, currently undated

@anmonteiro I tried your "string" require patch with re-natal—I think current setup there uses js/require a bit. My hope is to maybe dispense with that approach in favor of the new ns form capability. If so, I could better help validate this new feature.

Cool. I added an example to the ClojureScript repo too

@dnolen random thought that has been in the back of my head: given the amount of work that has been going into the ClojureScript compiler (and associated churn) what do you think about nightly releases (maybe through Maven snapshots)?

that way people who don’t know or want to build the compiler can try out things before they’re released

One related thought: We didn't seem to find bugs in the 1.9.665 release until it was cut. (Perhaps at least the "pre-release" pattern is useful.)

Travis supports triggering builds daily, so should be possible to configure this there

@anmonteiro that’s something we should sort out with Alex Miller I think

he’s on vacation, but when he returns we can bring it up

awesome

incorporating the CI work Mike has put together in the official repo would also be valuable IMHO

unless Alex has other ideas that include the current Clojure build system

CI is yet another discussion but that would probably not go very far - we have CI for Clojure

I think getting the existing CI to work for ClojureScript would be fine

and it would be great if we could build all dependents we maintain

core.async, test.check, etc.

thus why I think Nashorn test runner is important low hanging fruit

then Alex doesn’t have to go and sort out Node.js

all that said - I don’t think this means @mfikes Travis CI stuff isn’t useful

for example it would be great if that CI built community libs and tested them

Reagent/Om/Re-frame etc.

I wonder how hard it would be run tests inside a Docker container? Then it would be easy to maintain test environment for Cljs using Dockerfile at the repo

This is what I do with Cljsjs CI, I have Dockerfile with Java+Clojure+Boot+Node

If anyone is interested, you can now fork the ClojureScript repo, turn on Travis CI for it, and push potential patches to your fork and let the tests run there. I've been doing that lately (on an under-powered laptop).

@dnolen actually I hadn’t noticed that we have a .travis.yml file in the ClojureScript repo. Anyone with push access can set it up in http://travis.org

it’ll basically start building automatically

@anmonteiro yes

next step start build community libs against that and running tests

would be cool

Yes

Those tests can even run parallel so it should be possible to run them relatively fast

For now, I've been having the primary ClojureScript repo master branch commits pushed automatically over to https://github.com/mfikes/clojurescript where Travis runs the tests.

does having a clojurescript-oss org make any sense?

Would that somehow be useful for dependent libs. hmm... :)

I like the idea but what would you put there?

well I’m assuming we need travis scripts for OSS libs

we can collect them there?

If we use Travis or similar system, and the tests are triggered by cljs repo commits, it would be easiest to maintain them at the cljs repo

Om, for example, could have a .travis.yml in its own repo, But how could you get Om to run its tests for each ClojureScript commit / build?

regex replace on project.clj file 😕

I think what David has in mind is collecting some scripts to 1. build CLJS 2. clone those OSS projects 3. test them against pre-release CLJS

yes

correct me if I’m wrong

that’s going to inherit failures from those projects too if they’re in a failing state

daily or even per commit performance/output size etc. benchmark + graphs would also be cool

@anmonteiro this why cljs-oss would just be travis build scripts

can you not depend on other builds in travis?

That would be extremely useful for catching subtle regressions—especially around funky optimization work.

@dnolen Nope, Travis is quite simplified, builds are only triggered by commits or timed

then we could also make a github static page which shows build state

@juhoteperi does CircleCI support this?

...though both Travis and Circle have HTTP endpoint that could be called by other project

that can trigger the build

So, if a lib wanted to participate, it would PR it's built-relevant bits to clojurescript-oss. That would be real nice!

@juhoteperi that seems sufficient

Well, we can start with just testing the setup manually and setting up the trigger later

cljs-oss or clojurescript-oss for org name?

Shorter better

https://github.com/cljs-oss

So would the idea be that a .travis.yml clones ClojureScript, builds it, and then clones n other oss projects, and builds them against the just-built ClojureScript?

Or perhaps there could be multiple repos inside cljs-oss one per OSS project

I think you probably want one per OSS project

different projects may need different things

Makes sense. And the http endpoint mechanism can chain a ClojureScript clone & build based on a main repo commit, and each OSS repo can use the same mechanism to trigger their own.

One example: Planck is already set up to clone ClojureScript master, build it, and use it to build Planck, and then run its own test suite, all based on manually running its script/pilot script. It would be nice if this were automated inside cljs-oss as a repo

So one repo to build Cljs, and push it somewhere for oss project tests to use?

or people can just submit Travis scripts for building against master

so they don’t have pollute their own repos w/ this detail

Then each project needs to build Cljs again? Probably that is fine also

hrm why?

if that can be avoided - better

If each OSS project has it's own repo under cljs-oss

each with their own travis script

they should pull down the build jar

not rebuild

okay, but where do they pull the jar from?

wherever we want to put it 🙂

okay, that was what I was saying 😄

Notes and ideas: https://gist.github.com/Deraen/f3b25fd459b6af134c0836cde36fb3fb

I like it

@dnolen attached a rebased patch to https://dev.clojure.org/jira/browse/CLJS-1822

I still think this ticket is valuable for specifying a :file-min JS module to be processed under advanced optimizations

sounds good, but as the first iteration I would make it simpler, just one repo “cljs-oss/clojurescript”, which would on trigger do all the work at once, e.g. build new jar file and trigger all participating OS projects, links to results would be embedded directly in the git commit message (e.g using travis badges), so no need for collecting them into results repo

very useful in the React case for example, where they have different bundles for DEV and prod (this way we can provide a simple way for people to consume these modules without having to muck around with fake process.env namespaces)

@anmonteiro ok will look

@darwin so do you mean OS projects wouldn't have separate repos, or just the the output would not be commited to a repo?

yes, both, all maintainers of those “participation scripts” would collaborate together on a single repo, also because those scripts will likely trigger long-running tasks somewhere on the web (e.g. travis), you would just commit a list of links into the git message

without waiting

fwiw, I don’t think having the main thing do this is a good idea even in the first cut

individual libs might want to different completely unrelated kinds of things

the overhead doesn’t seem interesting to me at all either

people can sort out their own build

separate repos allows for e.g. only reagent test be triggered when there is commit to reagent repo, instead of all the projects being tested

yes

(though there are magical ways to avoid that... e.g. cljsjs CI scripts uses clojars API to check which packages have been deployed :D)

ok, I see your point, I wanted to avoid managing tons of repos which will have similar code in them

most repos will simply trigger travis build on their main repo with some custom env variable set to use specific clojurescript jar

I can imagine

ah, my idea is that the tasks clone the main repo, change the clojurescript version on project.clj/build.boot and run the test there

and no triggers back the main projects

that will duplicate some code that is present in OS project main repos, but it will make easier for us to do something with the results, e.g. push to a result repo

ok, but then I as a maintainer of libraryX will have to think about cljs-oos/lib-x and potentially update the thing whenever I touch travis-related stuff

hmm, touching the travis-related stuff (travis.yml) at main repo wouldn't affect cljs-oss/lib build?

though changing the test runner or such would indeed cause need to update cljs-oss/lib

I'll add a note about triggering build at the main repo with custom env variables, that could work, at least for those cases that run Travis/Circle

correct, I meant the test-runner stuff, but there are maybe some deps which are described in readme for developers who want to hack on the lib and test it, but this info is somewhere in .travis.yml in before_install or similar sections

I would have to keep them in sync with cljs-oos/lib, because that bootstrapping must be done there as well

hmm true

triggering builds over http endpoint requires API token for the target Travis project, will have to see how hard that will be configure for each project

that could be in travis secrets, that would the only thing someone super-admin of clojurescript repo could add

If cljs-oss/clojurescript is going to trigger OS project builds, we'd need token for each project

ok, I was speaking about the mono-repo situation and assuming the whole job will be done by travis, sorry, in multi-repo as you suggested, that probably has to be in secrets somehow, you don’t want random people on internet triggering your ci builds 🙂

e.g. if each of those project repos used travis, it can be a secret env encrypted in that particular repo for particular master repo

https://docs.travis-ci.com/user/environment-variables/#Defining-encrypted-variables-in-.travis.yml

but would the mono-repo run all the tests (and contain test script for every project) or trigger builds in other projects?

my current mono-repo idea, three branches master, builds, results : 0) builds are triggered by committing to builds branch with clojurescript SHA to be tested 1) uses docker to provide well-defined runtime environment 2) inside docker run a worker.clj script which accepts as a parameter SHA of clojurescript repo 3) this script will clone the clojure/clojurescript repo, build the jar and upload it somewhere (e.g. S3) 4) then there will be a list of libraries, each having their own job function, worker will call each of them to return some result (edn) 5) a job function for a project can do anything, but typical job function will trigger a build on its main repo (using travis secrets for access) 6) worker collects results from all jobs and prepares a commit message, this commit message will list all links, badge icons or whatever and commit it into results branch of the same repo

in this idea, new build is not triggered by changes to a library repo?

inspiration 1: here I have a docker file which tries to resemble travis-ci, I use it to run my travis scripts locally: https://github.com/binaryage/dirac/blob/master/test/docker/Dockerfile https://github.com/binaryage/dirac/blob/master/docs/hacking.md#running-tests-via-docker

@juhoteperi correct, I didn’t see that as a requirement, new clojure/clojurescript commits should trigger cascade of tests

Hm, it would be useful in cases where lib is broken and is fixed. But in most cases the libs are correct and Cljs shouldn't break them.

inspiration 2: here I have a health-check branch which triggers builds (I imagine builds to be similar to this), the commit message encodes specific versions for the build: https://github.com/binaryage/dirac/commits/health-check

It would also give faster feedback if something that is done to a lib is broken with new cljs

But probably not necessary

I see your point, let me think about it

in #2 of my list, the worker.clj script could be parametrized with more options, for example in addition to clojurescript SHA, with filter of jobs to run (e.g. a regex), library authors wanting to re-test their lib for hot-fixes could trigger it maybe?

But libs could check for this themselves, by always running tests against their default Cljs and then against latest cljs-oss jar

indeed

In that case we wouldn't get result back to cljs-oss

I think this can be communicated via github, each commit with broken stuff can have discussion, and lib authors can singnal that they fixed their libs

and it will be fixed on next trigger (or new commit) in cljs repo

I can imagine there will be broken things even not as a cause of cljs repo changes, tests might be flaky or some env problems

so broken tests are not something fatal, it will just signal clojurescript team that something didn’t go as expected and someone should take a note

Well, you should probably also have access to cljs-oss org so I sent you an invite

thanks, I’m in

in #4 the job functions can be a) shell scripts (for maximal flexibility) b) clojure functions (for convenience) - I assume most people will create their own clj (or sh) file and work on their own thing there, in clj case they could maybe reuse some helpers provided in some shared place

@anmonteiro left a question about CLJS-1822

if someone wanted to do something really heavy/crazy in their job function we would recommend to do it outside of this worker, maybe triggering their own http hook, travis build somewhere else

in #6 I would think that result edn will have just simple textual description for now, but later it could have some well-specified data (e.g. timings)

@dnolen right, no implicit magic there. We only use :file-min if the user specified an explicit override

ok

I added a test case that demonstrates this

@anmonteiro Did you test es6 module support after updating Closure? I'm having problems testing preprocessing, looks like babel is running okay, but looks like es6 module processing doesn't do anything

Oops we may have to back that out

@juhoteperi I'll have a look in a little bit

And write a regression test

I'll check with older Closure version to see if that helps

Do you have a simple example that I can use?

Works with v20170521

https://github.com/Deraen/saapas/blob/master/src/cljs/js/main.cljs.edn#L6-L9 + https://github.com/Deraen/saapas/blob/master/src/cljs/js/hello.js

Hah, nice. We definitely need tests for that stuff, so I guess this is a good opportunity

Just calling convert-js-modules with mocked js-module map (with source as string) should be enough

Hmm I thought I even wrote something like that..

Ah module_processing_tests only checks :commonjs

Should be easy to extend that with another module which uses :es6

Yeah, I know we have some for commonJS but I guess we need to exercise the ES6 code now that it's becoming more widespread

@anmonteiro I left another comment on CLJS-1822, lets keep the discussion there so it doesn’t get lost in the Slack aether

They rewrote the ES6 pass in Closure for this release, we may have to change something on our end

Sure

Answered both questions in the ticket

Re: preprocess symbol support, should Cljs catch exceptions from require and show warning or throw exception and stop the build?

Warning and continue would probably be most similar to current logic if the multimethod is not found

@anmonteiro thanks ok, makes sense will review one more time

But catching the exception would probably hide useful information about debugging the problem...

@juhoteperi ex-info

put the :cause in there

@juhoteperi also make sure to use load-mutex to lock the require

if you want to make a reusable require helper that does this - also cool with me

So I should throw ex-info with cause and additional information?

yes

meaningful error string, original :cause

Patch attached: https://dev.clojure.org/jira/secure/attachment/16981/CLJS-2143-1.patch

Hm, I should probably improve the error message a bit more

The warning messages could also be more specific about the problem, multimethod method not found, symbol doesn't point to a var/function, preprocess value is something else than keyword or symbol

One thing I was thinking about: Is processing files one by one good for all use cases? Are there some tools that need to process all the files in one go, like Closure CJS/ES6 processing?

to be honest I just can’t think of case where this is desirable

or even where in the JS world people do this - cross file analysis isn’t much of thing

@juhoteperi agree about better error messages here

What's the best way to add context to the warning (should there be single warning type or multiple?), add info the the extra map and use that in error-message multimethod?

@juhoteperi I don’t think we need to get fancy here, just assert and throw or wrap and rethrow w/ info if important

basically same as how we handle opts validation

@anmonteiro ok left one more comment on 1822

@dnolen hrm I just copied the tests over from what @juhoteperi had

let me try to use with-redefs

@anmonteiro https://dev.clojure.org/jira/browse/CLJS-2175

@dnolen attached new patch to CLJS-1822 that uses with-redefs

poking around I don’t see an obvious reason why Closure ES6 module processing would have stopped working

we set lang in/out and this looks OK

@dnolen some more windows fixes https://dev.clojure.org/jira/browse/CLJS-2176

there might be more coming

@dnolen I atteched a patch with broken test case

Works with previous Closure version

@juhoteperi patch is missing es6_hello.js ?

Oops

https://github.com/google/closure-compiler/commit/d24d175572ebf97b06e480f04c3f452bb6355afa this might be one related change

yes I’m discovering this by looking at the code 🙂

This is the feature set languageIn should have: https://github.com/google/closure-compiler/blob/master/src/com/google/javascript/jscomp/CompilerOptions.java#L3064

options.needsTranspilationOf(FeatureSet.Feature.MODULES) should be true as input featureSet has MODULES but output doesn't have

Oh, the ES6 transpilation doesn't happen at parse stage anymore

> This moves it from happening at the end of parsing into DefaultPassConfig, where it can be more easily reordered around other passes.

And Cljs code only calls parse

yeah I have it sorted here

fixed https://github.com/clojure/clojurescript/commit/e2fcea32ee234c3aaaf3ed0715401a8ce274eaf9

👍

@dnolen report-failure should probably be called after parse to check for parse errors?

hrm yeah

And if there are problems during parsing, module rewrite probably wont succeed

we should also probably reuse the get-js-root function

get-js-root doesn’t do exactly what we want here though

right I saw that

root has two nodes, first is externs next is the scripts

it does get the scripts child though

that was just a nitpick in any case

heh yeah, we might want to think a bit about making some sensible wrappers in front of all this imperative stuff

I see that Closure itself has gone down this route via pass factories, parseInputs which returns root etc.

but not for this commit 🙂

And if later want to support es7 features etc. we'll need to support other passes than Es6RewriteModules

actually there’s nothing to do there 🙂

the only reason we just rewrite modules here is to get a GCL namespace

advanced compilation will take care of all that other stuff

oh right, and then the ES7/8 features can be processed later

yep

this means we can get hot loadable JS files ASAP

w/o all the other stuff slowing things down

re: CI I think we should also set up AppVeyor

would make our Windows support first class

Yeah. Boot appveyour config could be helpful, it uses mingw for make and bash and I think there is grep available there, so tests could be run similar to travis config: https://github.com/boot-clj/boot/blob/master/appveyor.yml

@anmonteiro I like that idea - our Windows support has been spotty having CI catch obvious bad stuff would be huge

Previously preprocess multimethod could work with any values, e.g. strings, should that be still supported or can we presume it was only used with keywords?

Hmm, calling the multimethod always if the value is not symbol is even simpler than checking for keywords

we never said we supported anything but keywords, and it’s an alpha feature

and we may want to dump the multimethod later, so better just to close the scope

@dnolen report-failure should probably be called both after pars and after Es6RewriteModules, because the latter can also report problems: https://github.com/google/closure-compiler/blob/d24d175572ebf97b06e480f04c3f452bb6355afa/src/com/google/javascript/jscomp/Es6RewriteModules.java#L268

Hmm, in fact I'm not sure how those compiler reports work, are parse errors present even after Es6RewriteModules...

Looks like nothing will clear those errors, so I was mistaken and calling report-failure the last thing should be enough

But perhaps hasErrors should be called between parse and Es6RewriteModules, that is what Closure does: https://github.com/google/closure-compiler/blob/master/src/com/google/javascript/jscomp/Compiler.java#L722-L737