had anyone tried to use shadow-cljs to create chrome extensions?

I'm trying to create a developer tool there, but the connection doesn't work, it's trying to use a path relative to the extension, which ends up like this:

shadow.cljs.devtools.client.browser.js:811 WebSocket connection to '' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED

is there a way to force the URL for the websocket to be used as localhost instead of relative to page?

got it, closure defines 🙂

:closure-defines {shadow.cljs.devtools.client.env/devtools-url ""}

after more setup, new issue:

Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'self' blob: filesystem: chrome-extension-resource:". Either the 'unsafe-inline' keyword, a hash ('sha256-8QW/FL/JEeHNL6TDE0m/ACLlRnGNJ8rr574pBcCZ1hM='), or a nonce ('nonce-...') is required to enable inline execution.

that happens when shadow tries to reload the files

I tried adding "content_security_policy": "'unsafe-inline'", to the manifest, but chrome doesn't allow that =/

any ideas on how to get around that?

another issue regarding chrome, I need to compile and run things also in content scripts, and there it's pretty hard to read content dynamically, I'm seeing by the docs that shadow doens't use other compilation modes in dev, but I really need something like :whitespace, or anything that outputs everything in a single file, is there a way to do this in dev mode?

@lilactown this was a bug in the closure compiler. the latest shadow-cljs version has the latest closure so try that

@wilkerlucio :devtools {:devtools-url ...} is shorter or :devtools {:use-document-host false} should also work

hmm CSP would need the nonce. wonder if there is a way to get it

@wilkerlucio I changed the eval method in 2.3.23 and this seems to work then https://github.com/thheller/shadow-cljs/blob/master/out/chrome-ext/manifest.json#L10

took the default CSP rules from https://developer.chrome.com/apps/contentSecurityPolicy and added script-src: 'self' 'unsafe-eval';

only did the basic background.js test though. that loads fine and live-reload, REPL work

@thheller awesome! now the reload is working on background and on the devtool 🙂

but I noticed a few things after updating (I was in 2.3.21 before):

- my builds now seem to be starting one by one (I run 4 builds at once, so the startup got noticable slower)

oh. yeah I know why. will fix.

- the regular browser devcards build, it tries to use my IP directly and it didn't connected, I had to set the :devtools-url manually for it, while before it worked without setup

hmm maybe my boolean logic failed again, will check

thanks

and about the content-script page, you have any idea what we can do to work there?

what is that?

the loads seems to fail completely there, the content-script permission is weird

in the past I used to work in dev mode using :whitespace compilation for it so it loads on a single file

now it's just a bunch of:

what is content-script?

09:41:46.304 main.js:3027 GET  404 (Not Found)
(anonymous) @ main.js:3027
env.load @ main.js:3020
(anonymous) @ main.js:3065
09:41:46.306 main.js:3027 GET  404 (Not Found)
(anonymous) @ main.js:3027
env.load @ main.js:3020
(anonymous) @ main.js:3065
09:41:46.307 main.js:3027 GET  404 (Not Found)
(anonymous) @ main.js:3027
env.load @ main.js:3020
(anonymous) @ main.js:3065
09:41:46.307 main.js:3027 GET  404 (Not Found)
(anonymous) @ main.js:3027
env.load @ main.js:3020
(anonymous) @ main.js:3065
09:41:46.308 

content-script is when you want a chrome extension to inject code in the user page

on manifest.json is like this:

"content_scripts": [
    {
      "matches": ["<all_urls>"],
      "js": ["js/content-script/main.js"],
      "run_at": "document_start"
    }
  ],

hmm dev mode will probably not work for this

I wrote some about it in the past, because figwheel also didn't worked good for it: https://medium.com/@wilkerlucio/setting-up-figwheel-for-chrome-extensions-content-scripts-f1631d8e782a

we really need to control the global scope for the devtools stuff

I wasn't sucessful on making it work proper, but there is some info about loading there

just to clarify: this is for javascript that will be injected into any random page? meaning pages not written by you?

yes, correct

yeah :dev builds will never be reliable for this

:release is probably the only safe thing for this

I could add a :dev thing that only outputs a single file so the debug loader is not used

but live-reload or REPL will never work properly

you can get lucky and have it work but it won't be reliable or even safe

ok, it's reasonable, because currently it just doesn't work as dev at all, I would have to manually re-trigger a release everytime on every update

if we can it just re-compiling in a single file automatically, would help a lot

but I'm wondering, why can't the REPL work there?

during dev mode it assumes that YOU own the global scope

meaning it creates a bunch of global variables, eg. cljs, goog and so on

when injected into the page you do not own the global scope so the potential for conflicts is high

what are you trying to do?

its also a bit scary since you are technically giving the page you inject into access to your REPL

suppose your plugin injects into a page that is running a dev build

your code would replace the actual page code

and everything would explode

but on content scripts you kind have your own context, and you can inject those variable and reliably access then

I'm making a fulcro-inspect version that runs on chrome ext / electron

I thought the scripts are injected? ie eval'd in the page?

so the content-script will have to hook into the fulcro tools to send information to the remote ui (that will be in a chrome devtool/electron)

they are, but they have a special context

https://developer.chrome.com/extensions/content_scripts

Although the execution environments of content scripts and the pages that host them are isolated from each other, they share access to the page's DOM.

ah so they are isolated

ok thats fine then

ok then scratch everything I said.

sorry I wasn't clear before, it's been a while I read that, so yeah, separated envs, shared DOM

hmm whats the minimum of config I need for this?

will need a test setup for this otherwise I'd just be guessing

this should be fine on manifest:

"content_scripts": [
    {
      "matches": ["<all_urls>"],
      "js": ["js/content-script/main.js"],
      "run_at": "document_start"
    }
  ],

that will make the script run on every page

for compilation I currently have this:

:chrome-content-script {:target          :browser
                                        :output-dir      "shells/chrome/js/content-script"
                                        :asset-path      "js/content-script"
                                        :closure-defines {:devtools-url ""}
                                        :modules         {:main {:entries [fulcro.inspect.chrome.content-script.main]}}}

ok this should be easy

the :browser config is not really suited for this though

it appears that I must reload the chrome extension before it uses any updated code

or am I doing something wrong?

@wilkerlucio how important is "run_at": "document_start" to you? would document_idle be enough?

@thheller I don't know yet, was just setting up the env, maybe document_idle will be enough

is there a roadblock with document_start?

nah not really. currently the debug loader tries to be smart and checks if it can use document.write to load scripts

with document_start that is true

but that causes the scripts to load in your the ppage

not in the isolated context 😛

hehe, maybe we can have a flag for it

so we can override on content-script case

with document_idle the check fails and it then does the usualy async load + eval

yeah adding that currently

Hi! trying to do release build of project. Got this output for :target :node-script:

var shadow$provide = {};
#!/usr/bin/env node
(function(){
...

@zarkone which version are you one? I thought I fixed that

oh, sorry, forgot to checkout. I'm on something-recent-11

will now try to update, thanks!

Are macros supposed to work with shadow-cljs node-repl?

using macros or actually writing defmacro? using yes, writing no.

Latter.

Is there a way to write macros in node?

Not in node but to be used in node.

My use case is I'd like some computation to be done at compile time and statically defined in the app.

its the same deal as with any other macro for CLJS

they need to be written in CLJ

Right but I can't get it to work with node-repl.

you could write them in shadow-cljs clj-repl

you can't write them inside node-repl since that is a CLJS repl

Oh I see. So I need to run two REPLs?

It confused me because when developing my browser application I am able to write and use macros by default.

hmm? the rules for the browser are identical?

I was trying to follow this example: https://github.com/shadow-cljs/examples/tree/master/macros.

This is what happens: https://github.com/kennyjwilli/shadow-cljs-macros

not how that works

(ns shadow-macros.core
  (:require
    [shadow-macros.macros :include-macros true :as macros]))

@thheller thanks! got rid of this error.

this will try to load shadow-macros/macros.cljs AND the .clj file

since only the .clj file exists this fails

also, getting this with release build:

------ WARNING #1 --------------------------------------------------------------
 File: com/cognitect/transit.js:649:8

 variable module is undeclared
--------------------------------------------------------------------------------

------ WARNING #2 --------------------------------------------------------------
 File: com/cognitect/transit/impl/writer.js:256:8

 variable isObject is undeclared
--------------------------------------------------------------------------------

------ WARNING #3 --------------------------------------------------------------
 File: com/cognitect/transit/types.js:384:70

 variable Buffer is undeclared
--------------------------------------------------------------------------------

@zarkone yeah disregard those ... they aren't used anyways

I believe I also tried just :require-macros and that did not work either.

@kenny the macros example I made is how you should setup macros

meaning you write a .cljs file and .clj file with the same name

and the cljs file does a require-macros for itself

NEVER do dedicated .macros namespaces. thats just makes things more confusing

@thheller just haven't seen this warnings with boot-cljs and :advanced optimizations. Trying to figure out why some parts of my code in release silently doesn't work 🙂 Thank you!

@zarkone boot probably doesn't have those checks enabled

right, that might be the case: haven't seen default params enabled in boot for cljs compilation

@kenny taking my macros example you would write your macro in the app/lib.clj

and then use it from app/main.cljs

your example tries to skip the "additional" .cljs file but it is very important to have that

do not try to shortcut your way out of that

I had no idea that was a requirement.

it is not a requirement but it saves you from dealing with :require-macros or :include-macros all over the place

getting those correct is just a nightmare

I totally agree.

with the recommended setup you do :require-macros ONCE and only to require itself

no-one else ever needs to to know about macros at all

Wait so I don't even need to :include-macros then?

exactly

https://github.com/shadow-cljs/examples/blob/master/macros/src/app/main.cljs#L1-L2

clean and simple

@zarkone CLJS by default doesn't enable those checks either. its just something shadow-cljs does. if you want to find out why things don't work shadow-cljs check your-build might help

Yep that works. Wow, that clears a lot up. Thank you. I think this bit of information is important enough to include in the documentation somewhere.

@thheller oh, thanks a lot! will try it out

is that not documented anywhere else? this has been in CLJS for a couple of years

https://dev.clojure.org/jira/browse/CLJS-948

I don't think I've read anywhere about a "suggested" pattern.

I always wonder why people still do the old way

3+ years

are there any official macro docs?

There's this https://clojurescript.org/guides/ns-forms. But doesn't really suggest a pattern.

I'd even argue that makes things more confusing because it introduces you to all the different ways you can use macros 🙂

I guess its mentioned under Implicit Sugar

@thheller actually, I've known that I can get rid of :require-macros from recent defn podcast where you've talked about it 😄

but yeah super confusing that its not mentioned as the first and only thing

Exactly. That's what it needs. I don't have any reason to care about all that other stuff.

btw, was really good episode

it was fun! a bit all over the place but fun. 🙂

Is there a technical reason why the dummy CLJS ns can't be created behind the scenes?

That may be what your issue there is talking about?

No that's different.

the CLJS ns serves several purposes

you can write actual CLJS code there that you macros might want to use

and if you want to emit code that uses other namespaces the CLJS ns can require those to ensure they are loaded

since .clj files can't require CLJS files

Sure but I may not have either of those requirements.

technically it could generate a simple version if only a clj file exists yes

but thats kinda dangerous since ANY clojure namespace can then be loaded that way

in practice it is not a problem to create that simple .cljs file

That's true. What is the danger in being able to load and CLJ namespace?

confusion mostly

people might end up trying to use random CLJ libraries and wonder why they don't work

happens often enough as it is

That's fair enough

Thanks for the help!

@wilkerlucio the problem with a single output file is source maps. debugging without those is horror but they get waaaaaaaaaaaaay too large and slow

wonder if that could be improved somehow

@thheller yeah, not cool, after those changes is the regular :none working?

I'm playing with a coupe different approaches currently

a) output files like browser, load async over network

b) output files like brower, load via document.write

c) output one file

a) sucks because certain lifecycle events fire before the code is loaded eg. chrome.runtime.OnInstalled

b) sucks because chrome complains and you must reload browser extension manually on each code change

live-reload works though

c) seems best but source map gen is too slow 😞

there is this trick webpack and others do to just use eval + sourceURL

but that feels so dirty 😛

unfortunately https://groups.google.com/d/msg/google-chrome-developer-tools/7mONVX1IAYo/Psq3uIBoCgAJ was never fixed

or rather https://bugs.chromium.org/p/chromium/issues/detail?id=552455

gotcha, thanks for the clarification, this is a tricky one

How do prevent the clojure.spec exception from being thrown due to the non-conforming NS form from the JS string requires? I'm trying :jvm-opts ["-Dclojure.spec.compile-asserts=false"] in shadow-cljs.edn but it's not seeming to be working. is there a standard way to address this?

@jjttjj which error? invalid ns forms are never allowed?

the asserts can't be disabled since the ns form would be invalid

getting a massive spec error relating to an invalid NS form when I try to eval the NS form in the repl. It works fine outside of the repl connection. I assumed this was due to the (:require ["js-lib" :as js-lib"] ...) string form causing issues but I could be wrong

massive spec error where? are you trying to eval this in CLJ?

CLJS REPL should work fine for this but you might be trying to eval this in CLJ which won't work

doh! yup that was it, thanks!

@wilkerlucio I will think about this for a bit. you can get it sort of working now by setting

:chrome-content-script
  {:target :browser
   :output-dir "shells/chrome/js/content-script"
   :asset-path ""
   :modules {:main {:entries [fulcro.inspect.chrome.content-script.main]}}
   :devtools
   {:devtools-url ""
    :http-root "shells/chrome/js/content-script"
    :http-port 8020}}

but must use document_idle for now

:browser is complete overkill for this though.

:chrome-ext
  {:target :chrome.content-script
   :output-to "out/chrome-ext/background.js"
   :entry demo.chrome-bg}

this is better 😉

@thheller thank you very much for looking so quickly on it, I’ll try those as soon as I get home today

@wilkerlucio good timing I guess since I wanted to try some ideas for a chrome extension for shadow-cljs anyways 🙂

in the beginner's tutorial https://medium.com/@jiyinyiyong/a-beginner-guide-to-compile-clojurescript-with-shadow-cljs-26369190b786 It mentions you have to link the index.html to the target folder. Is there a way to do this as part of the build process in the shadow-cljs.edn file?

@thheller I'm still seeing the Uncaught TypeError: Cannot call a class as a function error in shadow-cljs 2.3.22

upped to 2.3.23, same issue

@thheller the version with the chrome updates is released?

I think that for now, I can just delete the _classCallCheck(this, Source); from the source of the npm module

is there a way for me to vendor the npm module outside of node_modules/? I see that I can redirect what it resolves to, but the only options I see are either :global or :npm. I'd like to stick it in another dir that I can commit to source control

@lilactown what was your error again?

oh wait .. it was graphql right?

that is not actually a closure issue

it is this one https://github.com/graphql/graphql-js/issues/1316

maybe not though .. the error was different https://github.com/thheller/shadow-cljs/issues/248

https://github.com/google/closure-compiler/issues/2822

this should be fixed in closure 20180506. maybe you have an older closure dependency?

as for npm fixing I would suggest taking the node_modules/graphql directory and publish it under @lilactown/graphql or so

then :resolve {"graphql" {:target :npm :require "@lilactown/graphql"}}

@theeternalpulse I don't recommend that setup. better use https://github.com/shadow-cljs/quickstart-browser

that just has a public/index.html that doesn't need to be moved or anything

@thheller sorry bothering, I can't find the updates with the chrome things, did you pushed those?

@wilkerlucio which updates? I didn't release anything related to that yet no

just got back home. will fiddle around with it some more now

ah, gotcha, no worries, I just had assumed you pushed, my bad

the config I posted doesn't work? the :browser one?

hehe I can crash chrome by pressing ctrl+shift+e trying to profile my background page 😛

Would it matter if I had clojurescript in my lein deps? I'm using the lein integration

Regarding the version of closure I'm usinf

just check with lein deps :tree

and yes that would matter

depending on the order of your deps

didn't tried yet, still not home, was just trying to check it before you might go sleep 🙂

Ok I'll check when I'm back at my computer 😁

when you announced the :target :chrome.content-script I though it was out

nah need to sort out the source map issue first

@wilkerlucio just pushed the basic version in master

:chrome-bg
  {:target :chrome-extension
   :output-to "out/chrome-ext/background.js"
   :entry demo.chrome-bg}

  :chrome-content
  {:target :chrome-extension
   :output-to "out/chrome-ext/content-script.js"
   :entry demo.chrome-content}

but no live-reload or REPL support yet

cool, but isn't the background settings different from the content-script one?

I mean, the background and devtools parts were working perfectly here

yeah you can stick with :browser for that one

but :browser does a lot of stuff that the chrome ext will never use

but if it works for you just stick with that for now

still just messing arround

hum, so the plan is for :chrome-extension for all, I would guess content-scripts would have a different setting compared to background/popups/devtools, I guess we will figure that soon

I need to check which capabilities the chrome ext itself has

seems like its a bunch of different ways to execute scripts

not sure they all run in the same runtime or are isolated from each other and so on

yeah, sometimes chrome forces you do make a lot of round before doing something, what you can do in background is different from what you can do on a popup page, that's also different from a content-script...

a different build for each seems like a bit of overkill but probably the only option

one build means only one live-reload/REPL

for compilation I think 2 should suffice, one for content-script, and another for the rest

because the content-script is the one that has a bit difference in security things

the other ones vary mostly on which chrome API's you can access, but that's not a compilation concern

I thought you wanted a REPL in the content-script?

and background

I do

but the REPL for background and pages is working already, isn't it?

I guess

bugs me a bit that the browser target does so much more stuff we don't need

if I can sort out this source map issue then all targets could technically use the simplified :chrome-extension

what kinds of things the :browser does that we don't need there?

generating manifest stuff

output wrappers

some code gen could be simplified

bunch of small stuff really .. but just the config alone could be easier

agreed

browser target code is 700+ lines

chrome ext could be 60 or so

less moving parts, easier to debug

if it wasn't for this damn source map issue I'd be done already

another option would be to rewrite the manifest.json

and just automatically add the scripts

> The list of JavaScript files to be injected into matching pages. These are injected in the order they appear in this array.

it already takes a list of files so I could update those

the problem is happening now is at initial load, or to refresh code after?

which problem? the only problem currently is no source maps

yeah, that one 🙂

so there are no source-maps at all?

:chrome-extension generates a single file

generating a source map for that single file takes 500ms+

since its huge

too slow for watch

and can also take quite a while to load in chrome itself

I think the manifest edit option might be best

difficult to coordinate though when running 2+ builds

don't want to deal with file locking and such

does that solve the problem on reloading code after?

not related to that at all. that all works without issues in theory. didn't enable it yet

the reloading problem is solved though by adding the script-src 'unsafe-eval'

that probably won't change since we need eval

yeah, and that's fine

would be nice to do some changes before releasing, but that is a deploy process problem

:flush (362 ms) with index map with only cljs.core

definitely not an option

:flush (9 ms) without any maps

@wilkerlucio master now has source maps and REPL but not live-reload doesn't work

but only the slow source maps so its not very practical

will do some more digging after sleep

thank you very much, I just got home, so time to try and hack 🙂

does shadow yet support continuously running tests? i know that’s not a … flavor… that people in the clj world like, but it would ease a lot of my coworker’s anxieties

looks like the default node target just spits out a file, then you can run it with node automatically with :autorun . i could probably setup some node watcher to keep reading that file for changes and rerun on change, but I dont want to duplicate effort

@lwhorton :autorun is specifically only for :node-test? what else do you mean?

i mean that if I run shadow compile test with :autorun true it will launch the node ./out/node-tests.js and spit the results, but then the node process dies… is there a built in way to keep rerunning that node command every time node-tests.js changes?

shadow watch test

but no the node process will exit every time

re-using that is currently not supported

that’s exactly what i was looking for, i’m just being stupid (forgot watch was a thing)

@thheller I tried adding the content script here now, it loads, but I'm getting errors: https://www.dropbox.com/s/hevvi7n2lps765i/Screenshot%202018-05-17%2019.56.34.png?dl=0

ps: I'm using the .23 version yet, not sure how to use it shadow from master

yeah thats the conflict stuff I mentioned. your injected code "replacing" the actual code in the page

so they conflict with each other

to try master just clone it and run lein install

ah, I'm using from the npm run version

I was reading the code on repo, I guess you do some scripting to make a release (because trying to install directly from git didn't work, there is no package.json on the root)

hmm?

you should just need lein install

don't mess with the npm stuff. it is not required.

ok

so I just override my local version, right?

yes

I see it still compiles one by one, would you like me to create an issue so we don't forgot about the paralell initial compilation?

yeah, forgot already 😛

done

[:chrome-content-script] Build failure:
invalid require
{:entry [fulcro.inspect.chrome.content-script.main]}
ExceptionInfo: invalid require
	clojure.core/ex-info (core.clj:4739)
	clojure.core/ex-info (core.clj:4739)
	shadow.build.resolve/resolve-require (resolve.clj:492)
	shadow.build.resolve/resolve-require (resolve.clj:482)
	shadow.build.resolve/resolve-entry (resolve.clj:496)
	shadow.build.resolve/resolve-entry (resolve.clj:495)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)

the :entry is to be a namespace, right?

just the symbol

ah

not in a vector

final version will have specs for that but didn't add those yet 😉

yeah, I tried to be smart and just move the code, that's what I get, hehe