Fork me on GitHub
#shadow-cljs
<
2018-01-09
>
thheller09:01:16

@hlolli so I thought about it a bit yesterday. the purpose of the :target abstraction is to capture as much as possible about the “final” product so it requires almost no post processing. In your case there should be a :target :chrome-extension where the config itself contains everything required to generate all the support files the chrome extensions want.

thheller09:01:37

:browser sort of fits but not exactly since you’ll never serve things from a webserver and such

thheller09:01:49

so some assumptions that makes are simply incorrect

thheller09:01:13

I only did a very basic chrome extension once to test but forgot just about everything

thheller09:01:22

shouldn’t be too hard to write a :target :chrome-extension though

thheller09:01:51

maybe it could even just look at the manifest.json itself and figure out what if needs to do from there

thheller09:01:54

I have some time later today, maybe I can take a look at this

thheller09:01:57

you could try writing it yourself but the docs are non-existent for that part so its going to be painful 😉

hlolli10:01:38

The only required file so far I've seen is https://developer.chrome.com/apps/manifest manifest.json, the :background :scripts would be needed to be automatically filled, but that can also be html file, doubt cljs users will prefer a html file, think we can ignore that option. There are some local storage file schemas :storage :storage-schema, not sure how to best include that into cljs config, but they are just json files. I can also be forgetting something, this is the first time I'm doing extension myself. But starting with the manifest I believe half the battle will be won.

hlolli10:01:54

I'm gonna give shadow-cljs/src/main/shadow/build/targets/chrome_extension.clj a try. I will ask here if I bump into problems.

thheller10:01:33

you can put it anywhere

thheller10:01:44

:target some.ns/process is supported

hlolli14:01:17

@thheller I wonder if it would make sense to add :escape-unicode false here https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/build/targets/browser.clj#L308 was useing icelandic chars for manifest metadata and they all got \u00xxx

thheller14:01:30

not sure? should be ok I guess?

hlolli14:01:18

ok, I'll create a PR tonight, another question, is it possible to access :module-hash-names from the state passed into this function. https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/build/targets/browser.clj#L277 (I'm not useing this namespace or file directly, but creating code deriving from it).

thheller14:01:38

:module-hash-names works by modifying :output-name for the module, so others don’t need to be concerned with it

thheller14:01:19

:output-name defaults to :module-id+`.js`, :module-hash-names turns it into <module-id>.<hash>.js

thheller14:01:18

not sure what you mean by accessing it?

hlolli14:01:23

ok so I misunderstood you yesterday 🙂 the data I'm looking for is basically cljs-runtime/*.js

hlolli14:01:59

and the :output

thheller14:01:52

the cljs-runtime files are written in that fn call

hlolli14:01:18

ok nice! so :source-ids is what I'm looking for

thheller14:01:26

:build-sources

thheller14:01:41

or when using :modules the :sources of each module

thheller14:01:49

:build-sources is all sources from a build

hlolli15:01:21

ah yes, these maps with depends-on and bunch of metadata. Ok understand, thanks!

thheller15:01:11

unfortunately the state is too large to print so there is no easy way to inspect it

hlolli15:01:19

or

[[:shadow.build.classpath/resource "goog/base.js"] [:shadow.build.classpath/resource "goog/debug/error.js"] [:shadow.build.classpath/resource "goog/dom/nodetype.js"] [:shadow.build.classpath/resource "goog/string/string.js"] [:shadow.build.classpath/resource "goog/asserts/asserts.js"] 
...elided
slashes to dots = real filename.

thheller15:01:36

those are the “source ids”

thheller15:01:05

each id you can use to (data/get-source-by-id state source-id)

thheller15:01:24

that map then contains all the source info for that particular source id

thheller15:01:49

:resource-name is the relative filename on the classpath

thheller15:01:00

:url contains the source location

thheller15:01:50

:file may also be present but only for actual file, not files from libs in .jars

thheller15:01:39

the data/get-output! fn returns the generated code for that resource

hlolli15:01:32

completly unrelated to chrome-extensions, there's nothing here in your setup preventing the possibility to provide cljs-compiler as gulp plugin spitting out unoptimized cljs code. Just an idea for later project.

thheller15:01:33

yeah how output is written is completely customizable

thheller15:01:10

this creates one singular big file since karma had issues loading files the “closure-way” when I tried it

hlolli15:01:22

yes, one big file is sadly what many modern js tooling relies on. Especially painful adding some cljs code mixed with typescript when one needs to wait a minute for clojurescript to spit 1 file in development.

thheller15:01:58

you can run shadow-cljs server which avoids spinning up a new JVM for everything

thheller15:01:07

which makes everything else “fast”

thheller15:01:11

when that is running we pretty much have better performance than typescript or others would have

thheller15:01:27

would be pretty easy to add on demand compilation of singular files if you don’t need a full “build”

thheller15:01:39

“build” always involves packaging things together and such

hlolli15:01:41

nice, I'm all for wirting js in one language. But there are many projects mixing babel, typescript and coffeescript in one. If cljs could be one of those with minimal tooling, it would be nice. In this case, start the server once and hide that from the user.

thheller15:01:04

shadow-cljs start runs the server in the background

hlolli15:01:34

yup, child_process.spawn() ideal I'd guess, just theorizing.

thheller15:01:34

but yeah most JS tooling is built assuming that you launch a new process for everything

thheller15:01:43

thats just not going to happen for CLJS

thheller15:01:08

if you want JS interop :npm-module may already solve that

thheller15:01:10

compiles CLJS in a way so any other JS tools can consume it (eg. webpack, node)

thheller15:01:14

made a teaser vid a while ago for create-react-app https://www.youtube.com/watch?v=BLDX5Twt2zk

hlolli15:01:34

I feel weird not knowing about this for so long 😄

thheller15:01:20

well for the first 3 years I basically never talked about shadow-cljs at all so that is probably my fault 😉

thheller15:01:55

been trying to blog more but I prefer writing code so I never get anywhere blog-wise 😉

hlolli15:01:20

good seacret, while others struggle 🙂

thheller15:01:07

figwheel is pretty deeply embedded in pretty much every template/tutorial these days

thheller15:01:32

not that easy to convince people to try something that does things differently

thheller15:01:48

but again I blame only me and the lack of documention/articles

hlolli15:01:04

yes, shadow-cljs is maybe a no-brainer to use for mixing cljs in js project, node platforms, browser plugins and probably Im forgetting a lot. Moveing divs with hot-reload, it wouldn't bother me useing relieable, compiler. After few years for figwheel, 99% of the errors I find like touching any part of my body blindfolded. Doesn't give me as much as it did in the past.

thheller15:01:50

well these days shadow-cljs pretty much does everything that figwheel does, just differently 😉

hlolli15:01:45

ok! How is the story with editors like emacs?

thheller15:01:41

emacs just works with cider, some people here use it

hlolli15:01:52

ah java is of course behind it

hlolli15:01:54

my last question, you see shadow-cljs and tools.deps (no lein or boot) a good combination, or would tools.deps be redundant?

thheller15:01:38

I will soon add support for tools.deps, similar to the current lein support

thheller15:01:56

that is that you will be able to manage your dependencies via deps.edn

thheller15:01:53

you can use tools.deps today but lose all the optimizations the shadow-cljs script does

hlolli15:01:02

yes would be nice, to be able to provide library that can be used by lein+boot and potentially lumo and planck. If there's a standardize deps format.

thheller15:01:12

clj -m shadow.cljs.devtools.cli compile foo is equal to shadow-cljs compile foo

hlolli15:01:38

but thats of course possible with maven .jar metadata

thheller15:01:07

I’m not totally sold on tools.deps for CLJS yet

thheller15:01:34

it is optimized for clojure after all and some things in CLJS are just different especially when integrating with npm and such

thheller15:01:24

might be better to start with a cljs-deps.edn than trying to integrate into deps.edn

mhuebert17:01:33

FWIW I’m really looking forward to being able to use git deps: https://clojure.org/news/2018/01/05/git-deps

thheller15:01:02

standardized deps format is definitely good but tools.deps assumes a classpath which is a JVM thing

thheller15:01:07

npm doesn’t have a classpath

hlolli15:01:15

Well then you have package.json as well, seems to be unavoidable to have two project files. Or integrade tools.deps into package.json. That's a different headace.

hlolli15:01:51

tools.deps gives me in lumo a nice way to resolve classpath for deps quickly and easily, nothing more so far.

thheller15:01:36

yeah, I have written my own thing for that https://github.com/shadow-cljs/shadow-cljs-deps

thheller15:01:57

if tools.deps had existed when I started I would have used that instead

thheller15:01:40

well written my own is an overstatement. I re-used what lein and boot use 😉

thheller15:01:25

you can run shadow-cljs node-repl if you need a quick REPL aware of your classpath, might not need lumo 😉

thheller15:01:48

but lumo is better if you want “scripts”

hlolli16:01:14

yes and portability. It's somewhat behind on dev tooling.

thheller16:01:18

well you can always use :target :node-script to build the “scripts” and run them directly. lumo just lets you skip the build step which is nice.

hlolli16:01:46

skips it completly yes if useing only bundled dependencies or native node functions.

thheller16:01:44

well you can always publish :node-script to npm (with a package.json to declare deps)

thheller16:01:27

but yeah for general scripting lumo wins

koz17:01:23

@thheller I pinged you several weeks ago regarding transitive NPM dependencies not working correctly when trying to import a JAR dependency built with shadow-cljs. You mentioned that you were aware of the issue but didn’t have the bandwidth to address it. Is this still on your radar? Happy to help in any way to get this implemented.

thheller20:01:42

@koz was that before :npm-deps? not sure I remember the issue

koz20:01:30

We did some work to port our front-end projects over to shadow-cljs, and while the NPM modules within that project worked great, we found that when we consumed those projects from another shadow-built project, the NPM modules coming from the dependent project couldn’t be resolved.

koz20:01:10

Haha it’s a bit hard to describe - let me know if that doesn’t make sense and I’ll try to provide more of a visual illustration of the problem.

thheller20:01:59

do you have the :npm-deps declared in a deps.cljs?

koz20:01:27

Yes, we did use :npm-deps inside a deps.cljs file inside of the dependent project

koz20:01:26

If it helps, we can try to put together a minimal test case

thheller20:01:48

you declared them but they dont get installed?

koz20:01:11

They get installed in that project, but not in any projects that depend on that project

thheller20:01:15

is there any public cljs lib that declares :npm-deps? I don't know of any

thheller20:01:38

I'm pretty sure I tested this and it should work

koz20:01:32

I’m not sure of any either - it’s possible that it works since we last tried it. We can try to set up a test case and put it up on a public repo if that helps.

thheller20:01:37

it helps to have a some lib available to test this with yes

koz20:01:10

Cool. I’ll put something together with the latest shadow-cljs version and will share with you once it is complete. Thanks!

Alex H20:01:26

Hm, I was just trying to use a stage-3 ES feature, but I don't think shadow-cljs is happy about it

Alex H20:01:37

is there any way to influence which babel plugins are used?

thheller20:01:19

babel is only used for npm stuff, ie. node_modules/*

thheller20:01:42

what’s the issue? which feature?

Alex H20:01:51

object spread

Alex H20:01:59

{...foo, etc, etc}

Alex H20:01:09

so babel isn't used for relative JS imports?

Alex H20:01:11

surely it is

Alex H20:01:34

:require ["../../js/foo.js" :as foo] sort of thing

thheller20:01:38

no, stuff from the project itself runs through the closure compiler to get :advanced DCE

thheller20:01:10

do you have a full snippet of code I can try?

Alex H20:01:33

it's really just a dumb trial:

Alex H20:01:39

function assoc(obj, key, value) { return {...obj}//...obj, [key]: value} } export { assoc }

Alex H20:01:57

and then the aforementioned :require in a cljs file

thheller20:01:17

that doesn’t look like valid code

Alex H20:01:37

well, it is valid stage3 code

Alex H20:01:38

as far as closure compiler is concerned, this is relevant: https://github.com/google/closure-compiler/issues/2303

Alex H20:01:45

so not yet supported, but maybe being worked on

Alex H20:01:15

anyway, my question, I guess, is whether it's possible to transpile things (slightly) with babel before feeding into closure compiler

Alex H20:01:36

(as in, out of the box with shadow-cljs, not talking manually here)

thheller21:01:12

I can add an option to disable the use of the closure compiler and just use babel

thheller21:01:18

using both is probably not a good idea

thheller21:01:21

babel generates a bunch of code the GCC doesn’t like

Alex H21:01:28

I was thinking doing the bare minimum in babel of transpiling to es2016 standard (even keeping the es2015 modules, etc), and then feeding that into closure compiler, so you still get DCE, etc

thheller21:01:51

well if the closure compiler supports it we should be fine

thheller21:01:35

looks like the commit made it in, unfortunately we can’t yet use the latest release of the closure compiler since CLJS breaks with it 😞

Alex H21:01:37

well, other than that the interop with non-npm JS files seems to work a treat, thanks for that

Alex H21:01:10

and if you are ever bored, having an option to feed non-npm JS files through babel (instead of closure compiler?) would be a nice addition to shadow-cljs (but certainly not a must-have)

thheller21:01:01

that is the only line that decided whether babel or gcc is used

thheller21:01:03

so easy to add

thheller21:01:36

ah doh no, not that easy

thheller21:01:48

the closure compiler needs to be able to parse the source file

thheller21:01:13

to extract the require/import/exports even if we pass it through babel after

thheller21:01:25

but that fails since it doesn’t recognize the code

Alex H21:01:38

how/why does that work with node_modules then, when they aren't closure-compiler compatible?

thheller21:01:40

but I have a sketch for passing code through babel FIRST

thheller21:01:28

I only parse the JS first to look at the AST for require/import

thheller21:01:41

so as long as its valid JS we are good

thheller21:01:20

I need to add a pre-process anyways to get support for typescript, coffescript, JSX, etc

Alex H21:01:11

well, it'd certainly be a nice feature

Alex H21:01:26

anyway, thanks again!

hlolli21:01:11

is it possible to rename the default output filename main.js, adding :output and :output-to in build options didnt made a difference.

hlolli21:01:59

not that I actually want to rename it, just want to get that name programatically, in the end.

thheller21:01:12

the module-id is the name so module :main is called main.js

thheller21:01:41

programmatically you get the name via :output-name

koz22:01:45

@thheller Sorry for the confusion before - just updated everything and it seems to work perfectly - thanks for all of your help!

koz22:01:59

Amazing job on this project.

hlolli23:01:22

So I believe I've succeeded with this, this can be polished (spec the manifest stuff) but everything put into :chrome-extension will be exported into manifest.json, only thing is to add: :target clj.chrome-extension/process (clj is just whatever source directory is on cp) https://gist.github.com/hlolli/265b9183566a4c5829d7ee355e3d0998

thheller23:01:18

runtime-bundled? (or (nil? optimizations) (= :none optimizations) (= :simple optimizations))

thheller23:01:34

:shadow.build/mode in the state is either :dev or :release

thheller23:01:45

is it safe to pass many scripts into “background”? though that was supposed to be one

thheller23:01:23

you can probably shorten this by just “inheriting” from the browser target

thheller23:01:26

ns clj.chrome-extension
  (:refer-clojure :exclude (flush require))
  (:require [shadow.build :as build]
            [shadow.build.api :as build-api]
            [ :as io]
            [clojure.set :as set]
            [clojure.data.json :as json]
            [shadow.build.data :as data]
            [shadow.build.output :as output]
            [shadow.build.targets.shared :as shared]
            [shadow.build.classpath :as cp]
            [shadow.build.modules :as modules]
            [shadow.cljs.repl :as repl]
            [shadow.build.targets.browser :as browser]))

(defn flush-crx-manifest [{:keys [build-options chrome-extension stage mode] :as state}
                          {:keys [chrome-extension optimizations] :as config}]
  (let [runtime-bundled? (or (nil? optimizations) (= :none optimizations) (= :simple optimizations))
        chrome-extension (:chrome-extension config)
        cljs-runtime-path (:cljs-runtime-path build-options)
        js-main (-> state ::modules/modules :main :output-name)
        js-filelist (mapv #(->> (data/get-source-by-id state %)
                                :output-name
                                (io/file cljs-runtime-path)
                                .getPath)
                      (:build-sources state))
        data (merge {:manifest_version (or (:manifest_version chrome-extension) 2)
                     :name (or (:name chrome-extension) "unnamed-shadow-cljs-extension")
                     :version (or (:version chrome-extension) "1.0.0")
                     :background {:scripts (if runtime-bundled?
                                             (into [js-main] js-filelist)
                                             [js-main])}}
               chrome-extension)
        manifest-name "manifest.json"
        manifest-file (data/output-file state manifest-name)
        manifest (with-out-str
                   (json/pprint data :escape-slash false :escape-unicode false))]
    (spit manifest-file manifest))
  state)

(defn process
  [{::build/keys [stage mode config] :as state}]
  (-> state
      (browser/process state)
      (cond->
        (= stage :flush)
        (flush-crx-manifest config))))