Fork me on GitHub
#shadow-cljs
<
2020-02-09
>
Gleb Posobin00:02:04

I am having trouble finding how shadow-cljs is doing it in its source. Could you point me please?

darwin00:02:31

I don’t know, the sources are too complex for me to understand just by reading them, if I had the project at hand, I would simply break in DevTools and look how shadow-cljs does it

darwin00:02:49

it is probably using just eval

Gleb Posobin00:02:44

Ok, thank you!

darwin00:02:33

or better put breakpoint into your reloaded fn and inspect the callstack

rberger09:02:34

I have attempted to make the chromex example/sample (https://github.com/binaryage/chromex/tree/master/examples/sample) work with shadow-cljs :chrome-extension I pretty much have the background page working (modulo some issue with the test-storage function). But the biggest problem is I can't figure out how to configure shadow-cljs to properly hook in the extension popup. I was just guessing it was related to the :browser-action but my guess is clearly not working. I am still pretty unwashed when it comes to cljs / shadow-cljs and chrome extensions. My attempt at this is at https://github.com/omnyway-labs/chromex-sample-shadow. The README details the issues. If anyone has any helpful hints, I would be greatly appreciated. It seems pretty close and once this is made to work, it could be a good template for building chrome extensions with shadow-cljs.

darwin12:02:59

@rberger I will look into this today. Will try to use your project as my start point and try to make it work.

rberger21:02:54

Thank you! Very much appreciated

Gleb Posobin16:02:27

The problem is that the outputs should be specified in manifest.edn, not in shadow-cljs.edn.

coby17:02:15

Hello, so I'm trying to containerize a basic ClojureScript/shadow-cljs development environment using Lando (https://docs.lando.dev). So far I've got the dev server serving static HTML/JS as it should be, but the HMR server only ever reports {:type :client/stale} . Even when I ssh into the container and curl it directly:

$ curl 'localhost:9630/ws/worker/dev/ac9d0f31-11a6-4c6b-9c75-a6f6a5ed1756/8dcdadc5-c951-4188-9c08-c5881f658fe7/browser' -H 'Accept: */*' --compressed -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Extensions: permessage-deflate' -H 'Sec-WebSocket-Key: kC/iKKLLhR2/TcOVtWVO7Q==' -H 'Connection: keep-alive, Upgrade' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Upgrade: websocket'
{:type :client/stale}
Not sure what to poke next to troubleshoot. Thanks!

coby17:02:53

I should clarify that HMR works perfectly when I run shadow-cljs watch dev directly on my machine. So something about the container environment is keeping it from seeing and/or reporting fresh compilations.

Gleb Posobin17:02:19

@rberger I made a pull request which should make things work.

rberger21:02:07

Thanks, I woke up thinking that at least some of the configuration needed to be in the manifest.edn. The comments in https://github.com/thheller/shadow-cljs/issues/279#issuecomment-392007641 made it seem like some or all could/should be in shadow-cljs.edn.

thheller18:02:33

@ctamayo :client/state means that the js files you loaded in your HTML were not produced by the watch you started. so I'm guessing you are serving old files or have some cache enabled that serves them

coby18:02:26

@thheller is that determination made by proc-id? Here in ws.clj?

(not= proc-id (:proc-id worker-proc))
      (send-ws-error :client/stale)

coby18:02:26

If so, that could mean that the newer watch may not be generating new files at all, correct?

thheller18:02:29

please do not assume that there is a bug in shadow-cljs regarding this

thheller18:02:41

this is for sure a bug in whatever you are doing

thheller18:02:53

watch puts files in a dir

thheller18:02:56

load those files

thheller18:02:58

thats about it

thheller18:02:37

load it with whatever webserver you want but you need to load exactly those files

thheller18:02:54

since you tried it with curl you can see the ID it sends

thheller18:02:06

I presume you got those from the outdated generated files

coby18:02:24

no no no, this is definitely a bug in my code. I'm just trying to understand how stuff works. 😄

thheller18:02:49

like I said ... watch produces files in a dir, load those files. nothing else to it

thheller18:02:04

if you containerize stuff its easy to mess up paths and stuff

thheller18:02:19

or loading files from the container instead of the actual fs etc

thheller18:02:15

don't know anything about the lando stuff so can't make suggestions on that front

thheller18:02:38

it could just be a bad browser cache

coby18:02:00

right, so what I'm trying to determine is what "paths and stuff" are likely to get messed up, so I know which direction to go in. So let me rephrase: it's possible that something about the paths and stuff make it so that the watch is not actually producing new files, or not putting them in the right place?

coby18:02:09

by "bad browser cache" do you mean the browser itself could be erroneously caching the result of a previous request to the WebSocket?

thheller18:02:05

which webserver do you use?

thheller18:02:28

the websocket is not involved in this issue

thheller18:02:03

the issue is that the JS you are loading is outdated. it thus has an outdated websocket URL, which the server is telling you.

thheller18:02:44

all you have to do is look at your :output-dir and make sure that is the actual files you are serving

thheller18:02:15

just delete the entire dir and start the watch

thheller18:02:23

if the file re-appear you have the right folder

thheller18:02:29

if they don't you are not looking at the correct place

coby18:02:41

ah, I see. Things are starting to make more sense. Thank you!

darwin18:02:55

I just tried to compile the example chromex extension with shadow in release mode, backgropund page and content script work, but popup (browser action) was generated without dependencies This is the compilation output with --pseudo-names: https://gist.github.com/darwin/a2ce8ee9079f78842f62e9e89dfc6615#file-popup-js-L96 $chromex$ext$runtime$connect_STAR_$$ is missing

thheller18:02:27

it has been a while since I looked at the chrome-ext target

thheller18:02:59

but one thing to remember is that it defaults to using code-splitting

thheller18:02:08

so you are likely just not loading the shared file

thheller18:02:35

there should be a shared.js in the output-dir

thheller18:02:47

in outs that use a html file you need to load both

thheller18:02:00

ie. shared.js + popup.js

thheller18:02:23

otherwise the :output-type controls how the output is structured

thheller18:02:32

ie. for content-scripts its just one file with no splits

darwin18:02:37

aah, let me try it, btw. interesting is that in dev mode this worked event without shared.js

darwin18:02:45

I would expect both modes to be broken

darwin18:02:55

ok, great this fixed it

darwin18:02:37

so last standing issue is with source maps, devtools does not allow loading .map files from chrome-extension://…/something.map URLs

darwin18:02:52

I had to start dev-server and serve those smaps separately

thheller18:02:56

I'm pretty sure that works absolutely fine for me

thheller18:02:01

are you sure you have the right paths though?

darwin18:02:25

ok, let me try that it again

thheller18:02:38

can't tell which mods you did to the output.clj

darwin18:02:28

I was able to reproduce the source map issue after removed that custom hack. In current Chrome and also in Chrome Canary. You have to open extension’s background page to get spam of `

darwin18:02:16

like this, source maps don’t work

darwin18:02:52

the urls are accessible when I open them in a separate tab

darwin18:02:23

chrome canary hints on “chrome-extension://” is the issue

thheller18:02:00

why is that a problem?

thheller18:02:08

I get the same error though

thheller18:02:16

look like valid maps to me

darwin18:02:45

DevTools don’t see those source maps, so for example clicking on a link to source code in console does not work, it shows generated .js file , not cljs file as expected

thheller19:02:22

I would not expect anything to work if it tells you that parsing them failed

thheller19:02:33

so I'd first try to figure out WHY it failed to parse

darwin19:02:15

that is just a misleading message from current Chrome - Chrome Canary has it improved and communicates what went wrong

darwin19:02:00

this is how it looks in current chrome canary

darwin19:02:57

the source maps are perfectly ok, just the code responsible for loading sourcemaps for chrome devtools is picky about URL scheme

darwin19:02:20

when I use my prefix-hack to load the same source maps via dev-server it works as expected

thheller19:02:41

but why does it serve the file perfectly fine if you open it?

thheller19:02:53

the files should be served by the chrome-ext

thheller19:02:03

it makes no sense to it to talk to http

darwin19:02:27

we would have to look how source maps download is implemented in Chrome DevTools

darwin19:02:10

maybe they are calling some internal chrome API which has some security hardening and uses different code path than is used when you load page in normal tab

darwin19:02:14

if that makes sense

thheller19:02:35

maybe it needs to be allowed in the CSP in the manifest?

darwin19:02:58

this should not be related because DevTools is not the same context as extension itself

darwin19:02:27

this is the originating place for the error message: https://github.com/ChromeDevTools/devtools-frontend/blob/master/front_end/sdk/SourceMap.js#L232 I’m digging further

darwin19:02:51

btw. this is long-standing issue, I never found a good solution for it, that is also why I advocated inlined source maps as option in shadow-cljs

thheller19:02:25

I fairly confident that it worked the last time I worked on this

thheller19:02:40

but that was quite a while ago so chrome probably just messed something up

thheller19:02:51

I'd rather not work around chrome bugs

thheller19:02:07

it clearly has the right url and knows how to serve it

thheller19:02:27

it does serve it with text/plain though. maybe it doesn't like that for some reason?

darwin19:02:32

so it depends on host implementation of that loadNetworkResource fn, and I guess that is picky about that chrome-extension:// URI

darwin19:02:37

that net::ERR_UNKNOWN_URL_SCHEME looks like coming from C++

darwin19:02:21

at least I don’t see any trace of it in DevTools codebase itself

kaosko21:02:04

I have a leiningen multimodule build with shadow compiled reagent ui. now, shadow modules are meant for runtime modules and shadow doesn't seem to understand lein-modules. Is there a way to include cljs sources from a different lein-module? part of the ui is supposed to be deployed as a lib and shared between two different applications

kaosko21:02:22

the best I have for now is just to set the src path to my shared module. It's not too bad but requires everybody else to checkout the same layout even though they may not work on all of the other parts

thheller21:02:17

don't really know that lein-modules are

thheller21:02:25

as far as shadow-cljs is concerned all it needs is a classpath

thheller21:02:15

> part of the ui is supposed to be deployed as a lib and shared between two different applications

thheller21:02:47

that also makes no sense to me. you mean the sources should be shared right? compiled output can't be shared between builds

Gleb Posobin21:02:18

(sorry, looks like many questions about extensions today)

kaosko21:02:52

thanks @thheller. lein-modules are https://github.com/jcrossley3/lein-modules but perhaps I need Leiningen checkouts instead. Yes, we'll I don't have the share lib/module yet, but yes I figured easiest just to share the source

Gleb Posobin21:02:48

Tldr: content scripts are hot reloaded when code changes, but on page refresh the old content script is loaded by chrome. How can I make my content script fetch the last script from shadow-cljs and execute it on page load?

thheller21:02:22

I really can't remember much details about chrome ext stuff

thheller21:02:54

but pretty sure that some things can't be worked around since its just how chrome exts work

thheller21:02:10

ie. load things once and then never look at the FS again

thheller21:02:26

I vaguely remember that you can reload the chrome ext via the REPL

thheller21:02:43

which will make it reload things from disk

Gleb Posobin21:02:37

Yeah, but if that script that is loaded only once says to load the main content script from shadow server, that file on disk doesn't need to be refreshed by chrome. I just had trouble finding where shadow-cljs does hot reloading in its sources.

thheller21:02:19

what does that have to do with anything? you said that works?

Gleb Posobin21:02:46

That should work, but I don't know how to do that.

thheller21:02:56

you are confusing me 😛

thheller21:02:38

this really isn't all that complicated. shadow-cljs writes files into a folder.

Gleb Posobin21:02:44

Basically I want to do hot reloading not only on file changes, but also when the browser loads the page and executes the content script.

thheller21:02:54

whether or not chrome will read those files I cannot control

Ivan Koz21:02:39

hi guys how to get release-snapshot?

thheller21:02:02

what is a release-snapshot?

Ivan Koz21:02:34

was it ever released?

thheller21:02:03

name changed to build report 😉

Ivan Koz21:02:29

i'm a bit new to whole JS front thing, kinda confused =)

darwin21:02:39

ad that chrome ext caching problem - I proposed a kind of workaround, where content script code would init with a check for stale source code situation and did one initial hot-reload pass if stale content was detected but I guess this smells like a lot of hairy work

darwin21:02:02

maybe having content script impl setup as a dynamic module and doing intial dynamic module load every time on init could also fix it

thheller21:02:48

my guess would be that nothing will work

thheller21:02:02

I'm just assuming here that chrome loads the file from disk once and caches it

thheller21:02:17

so unless there is a way to tell it to reload that file from disk

darwin21:02:24

that dynamic module would be loaded via dev-server

thheller21:02:34

any magic other method that tries to read that file would also get it the cached version

darwin21:02:51

this is not how it works

darwin21:02:11

it is cached by extension subsystem, that has little to do with regular network cache in chrome

darwin21:02:41

and you could always add cache buster when loading module dynamically

darwin21:02:05

in case regular network cache would be a problem (a separate one)

darwin21:02:05

I think I could try to implement example of this tomorrow, content script would only contain dynamic module load, and that dynamic module would be the main content script implementation

thheller21:02:11

you could just REPL into the thing and (js/chrome.runtime.reload)?

darwin21:02:28

hmm, that could work as well, I wasn’t aware of this API

thheller21:02:03

maybe even just call that as part of the reload lifecycle?

thheller21:02:13

dunno exactly what it does

thheller21:02:26

maybe there are other methods like that

darwin21:02:54

@posobin this is worth trying

thheller21:02:57

also how do you inject the content-script in the first place?

darwin21:02:37

I don’t understand the question, chrome extension injects it

thheller21:02:04

you can inject it via code

thheller21:02:22

which might take the latest code always and not have that cache issue?

darwin21:02:29

dunno, but it is another promising thing to try

darwin21:02:57

definitely executing it via code should be possible solution

darwin21:02:38

if you are able to fetch fresh version via dev-server or similar side-channel

darwin23:02:12

I just tried the js/chrome.runtime.reload and it is not a good solution it blows extension state, including background page and others

Gleb Posobin23:02:45

The runtime.reload is the same as reloading from the extensions setting page.

Gleb Posobin23:02:22

I am still trying to understand how shadow hot reloads the file.

darwin23:02:00

I tried that idea with the dynamic code eval, I’m able to fetch it from dev-server, but I ran into issues with goog.provide, it throws when stuff gets redeclared

darwin23:02:51

@posobin look into shadow/cljs/devtools/client/browser.cljs

Gleb Posobin23:02:03

Yeah, was looking at that.

darwin23:02:18

it uses js/goog.globalEval

darwin23:02:22

I think we could be able to hook do-js-reload and on content script init replay all reloads since last extension reload

Gleb Posobin00:02:31

I haven't figured out the resources ids so far.

Gleb Posobin03:02:07

Whew, I have finally done it.

(defn refresh-source []
  (let [this-file
        {:resource-id [:shadow.build.classpath/resource "ext/content.cljs"]
         :module :content-script :ns 'ext.content
         :output-name "ext.content.js"
         :resource-name "ext/content.cljs"
         :type :cljs
         :provides #{'ext.content}
         :warnings []
         :from-jar nil}]
    (shadow.browser/handle-build-complete
      {:info        {:sources [this-file] :compiled #{(:resource-id this-file)}}
       :reload-info {:never-load #{} :always-load #{}
                     :after-load
                     [{:fn-sym 'ext.content/loaded
                       :fn-str "ext.content.loaded"}]}})))

Gleb Posobin03:02:59

Init function has to call this with some timeout (I use 500ms) to let the shadow cljs make the connection I think, not exactly sure why.

Gleb Posobin03:02:04

content.cljs is the name of the content script file, loaded is the function that you want to run when the content script loads (not necessary to add any tags to it).

darwin16:02:54

@posobin cool, but I wonder if this works more generally if you have multiple cljs files in the content script output

Gleb Posobin16:02:06

You can specify multiple files there.

Gleb Posobin16:02:42

Or do you want to get the list automatically?

Gleb Posobin16:02:38

For that you would have to query the shadow cljs server for the list of changed files first, I have not seen such methods in its sources.

darwin16:02:29

here is what I did: 1. wrap js/shadow.cljs.devtools.client.browser.do_js_load and record all reloaded sources there 2. replay recorded sources on intial load in content script the problem is “where to record that data”, I ended up sending it to background page and requesting it back on initial load

darwin16:02:43

but your solution seems to be more straightforward

darwin16:02:58

people just need to maintain that list of files to refresh

darwin16:02:01

which is ok

Gleb Posobin16:02:37

How did you do that?

Gleb Posobin16:02:51

I haven't figured out how to change the shadow sources.

darwin16:02:15

something like this

darwin16:02:08

btw, hooking that fn just for printing sources for your method could be helpful as well

Gleb Posobin16:02:26

Yeah, I was messing with the compiled output...

Gleb Posobin16:02:30

Basically spent the whole day yesterday on that...

darwin16:02:36

an alternative would be a macro which could give you all relevant namespaces in the content script project and you would synthetise resources (sources) from it

Gleb Posobin16:02:39

And how do you request it on init?

Gleb Posobin16:02:35

Also how do I patch the shadow server?

darwin16:02:51

why do you need to patch shadow server?

darwin16:02:59

I use js sources recorded on client side

darwin16:02:09

I basically replay all hot-reloads

darwin16:02:20

it does not do a server roundtrip

Gleb Posobin16:02:30

Ah, you record it in background.

Gleb Posobin16:02:34

Not on the server.

darwin16:02:03

yes, do-js-load works on client side, it has :js with fresh source code

Gleb Posobin16:02:33

Yes, I understand that. I thought it was doing the post-message to server.

darwin16:02:53

this is request to reload sources on init load, didn’t finish this part properly background page then replies with all recorded sources and I feed it into do-js-load

darwin16:02:53

potential optimization would be to prune all sources with the same resource it and use only the last one

darwin16:02:46

but in this form my code does not need to understand sources structure, it just records it and replays it on init load in the same order

darwin18:02:47

I have it almost working, just struggle with Error: Namespace "..." already declared.

darwin18:02:06

it looks like when I load the same js source it complains and when shadow-cljs does it, it works

Gleb Posobin18:02:07

Wait 500ms before requesting.

Gleb Posobin18:02:29

That solved it for me.

Gleb Posobin18:02:37

This exact problem.

darwin18:02:46

wow, thanks, that fixed it

darwin18:02:31

as I wrote above, it could be a bit smarter and filter out all repeated resources and use only the most recent one for each

darwin19:02:15

I’m done for now, will leave it in experimental branch for people interested in something like this

darwin19:02:39

if you discovered some better way how to do this, please let me know, thanks

thheller21:02:31

I really don't quite follow anymore what you are doing here but the solution looks crazy to me. there has to be a better way.

thheller21:02:00

to recap the problem is that a content-script does not load the correct source from disk once it has been injected right?

thheller21:02:21

ie. did anyone bother too look at the actual content-script.js to see if it gets updated correctly?

darwin21:02:56

the problem is that content script lifetime is dependent on the hosted page - if you refresh the page in browser, the content script’s javascript context is destroyed and new instance of content script is injected into the page - unfortunately the extension subsystem injects cached version, because extension itself wasn’t refreshed (the background page is still running and want it to stay)

thheller21:02:43

yes, that is what I meant by not loading the correct source from disk

thheller21:02:58

so it keeps the source cached somewhere

darwin21:02:01

when you have content script injected and don’t refresh the hosting page, and when you modify sources, shadow-cljs correctly hot-reloads changes, but these changes get lost on next page refresh

thheller21:02:22

slow down for a second please ... don't bring hot reload into this

thheller21:02:34

answer that simple question .. is the file updated on disk?

thheller21:02:38

did anyone ever check?

darwin21:02:55

yes, the file is updated on disk

darwin21:02:10

we both checked independently

thheller21:02:24

and did you try the programmatic inject?

darwin21:02:59

no, that sounds like too much work, I want declarative inject to work

darwin21:02:12

if I understand it correctly, I would have to implement some process watching pages being opened and inject code depending on what page is opening

darwin21:02:35

then I would probably need my content script to be single file, which would limit build config, I guess

thheller21:02:22

the way this is currently handled is already problematic so there needs to be a better way anyways

darwin21:02:11

maybe I could put some time in and fix this in chrome itself: https://bugs.chromium.org/p/chromium/issues/detail?id=104610#c1

darwin21:02:37

the solution would be for chrome extension sub-system to watch for modified files on disk in dev mode

thheller21:02:24

it is trivial to fix this by just generating a little bit different code

thheller21:02:49

let me try to setup a simple reproduce for this first .. I still don't quite understand what this is all about

thheller21:02:46

ah right I already have this setup ... in the chrome-ext build 😛

darwin21:02:07

have a link? I could test it as well here

thheller21:02:24

in the shadow-cljs repo

thheller21:02:35

:chrome-ext build was commented out

thheller21:02:58

I'm sorry but that setup is so far away from my usual setups that I cannot follow it at all

thheller21:02:59

I don't like splitting things into multiple source paths like that when one would do 😛

thheller21:02:12

the namespaces already describe what they do perfectly fine

darwin21:02:53

IMO, if you perfectly understand what your build tool does, then yes, otherwise it is just defensive stategy which won’t hurt 🙂 this layout was created by me in leiningen + cljs-build plugin times

Gleb Posobin22:02:11

@thheller steps to reproduce what's not working: load your extension example in browser, open any page, "▶️❤️◀️" will be printed in the console, change that string in content.cljs, reload the page, "▶️❤️◀️" will still be printed instead of a new string.

thheller22:02:39

yeah I have that

thheller22:02:18

already have a fix for it ... just looking for others 😛

👍 4
Ivan Koz21:02:47

i'm not quiet understand what am i asking, but how do i specify js deps while shadow-cljs managed by lein? webjars?

thheller21:02:17

js deps are handled via npm, so package.json. shadow-cljs does not use/support webjars.

Ivan Koz21:02:18

so for a fullstack app that will be project.clj for backend deps and project.json for front?

thheller21:02:03

js deps are managed in package.json/npm

thheller21:02:32

CLJS deps are handled by shadow-cljs.edn or project.clj or deps.edn (whatever you chose)

Ivan Koz21:02:15

alright, i'll research more on that, thank you Thomas