Clojurians Log v2

Generally I'd look into the contents of installed package. Sometimes they come precompiled, meaning that there's JS artifact + assets artifacts. If that's your case, I'd then try to require JS artifact and load asset (CSS) artifact.

Yeah that is what I intended to do, but I am not certain how and where I do that.

I guess just requiring alignment-viewer-2 in cljs fails shadow build?

Yeah, that ends up complaining that there is an illegal character in the scss file required here: https://github.com/sanderlab/AlignmentViewer2.0/blob/master/src/common/MolecularStyles.ts#L6

So I wondered if I can specifically direct shadow not to compile that package but instead use their pre-compiled stuff

looking at package contents, it comes with multiple artifacts try to require alignment-viewer-2/dist/standalone/alignmentviewer, and copy alignment-viewer-2/dist/standalone/alignmentviewer.css into your web server's public directory and load that CSS file as a normal link tag in HTML

Ah! Thanks! That really demystified external requires for me! Turns out the package still produces some error that is now virtually impossible to trace in the minified JS 😕

Uncaught TypeError: I.current is null
    useState alignmentviewer.js:4
    AlignmentViewer alignmentviewer.js:4
    React 10
    flushWork scheduler.development.js:265
    performWorkUntilDeadline scheduler.development.js:534

hmm, standalone artifact is probably a wrong way to use it, it's essentially a bundle that already comes with everything, including React, this can be problematic also looking into other artifacts, they all require scss files sounds like there's no easy fix to this, unfortunately

that last error looks to me like wrong usage, as in your code using it wrong in some way

but yes, shadow does not support processing css, let alone scss. you can setup webpack for that and use :js-provider :external instead https://shadow-cljs.github.io/docs/UsersGuide.html#js-provider-external

So I looked into the Webpack thing, and if I understand this correctly: this

:js-options {:js-provider    :external
                              :external-index "frontend/target/index.js"}

<script src="/js/libs.js"></script>
    <script src="/js/compiled/app.js"></script>	

Error: Dependency: alignment-viewer-2 not provided by external JS. Do you maybe need a recompile?
    shadow$bridge libs.js:2
    <anonymous> shadow.js.shim.module$alignment_viewer_2.js:3
    globalEval app.js:434
    evalLoad app.js:1405
    <anonymous> app.js:2014

TypeError: shadow.js.shim.module$alignment_viewer_2.FastaAlignment is undefined
    unknown_client$core$mount_root core.cljs:86
    unknown_client$core$init core.cljs:93
    <anonymous> shadow.module.app.append.js:4
    globalEval app.js:434
    evalLoad app.js:1405
    <anonymous> app.js:2016

How do I find out what the external JS provides?

just look at the frontend/target/index.js

So the return of require("alignment-viewer-2)" is undefined then, right?

var ALL = {};
ALL["react-dom/client"] = require("react-dom/client");
ALL["react-dom"] = require("react-dom");
ALL["react"] = require("react");
ALL["alignment-viewer-2"] = require("alignment-viewer-2");
global.shadow$bridge = function shadow$bridge(name) {
  var ret = ALL[name];

  if (ret === undefined) {
     throw new Error("Dependency: " + name + " not provided by external JS. Do you maybe need a recompile?");
  }

  return ret;
};

shadow$bridge.ALL = ALL;

Ah nevermind, trying to run webpack again fails this time because loaders are missing, so probably it never worked in the first place. Which is weird, because libs.js does contain code of the external thing

I'd expect that some webpack configuration is required to make the scss work

the packages may also do some async imports, which will make webpack treat the whole build as async. that means that the dependencies become available "too late", as in only after the shadow bridge thing wants to access them

If that is the case, is there still a way to salvage this?

in that cause you could try :target :esm with :js-provider :import https://shadow-cljs.github.io/docs/UsersGuide.html#_third_party_tool_integration

but that unfortunately also means that the whole output goes through webpack, not just the JS parts

that imposes some limitations and REPL/hot-reload are basically non functional after ...

Geez the last time it was such a hassle to use some code people have put on the internet was when I was trying to use an academic python package published more than six months ago. ^^

I don't think many people outside the original authors use this, given that npm says "6 weekly downloads" for this package 😛

most packages tend to sort out the css situation before publishing to npm

ReferenceError: shadow$bridge is not defined

I figured out the SCSS stuff

but whaat?

yeah thats the async load problem then I'd suspect

> academic I feel that's the root cause. :D I have worked in academia (and in genetics as well) enough to experience it in full.

ffs

If I were you, I'd definitely try to use their older version if its feature set and performance are enough for you.

From the looks of it that will basically require tearing apart the code. So maybe I will learn JS proper after all.

Are there any ways to connect to the ws server from a esm build? I’m build an esm module for a content script of a chrome extension.

not sure. chrome extensions these days are fairly locked down. if you are allowed to open a websocket connection it should work

by default it will try to connect to the page host through, which for chrome extensions won't work. so you set :devtools {:use-document-host false} in the build config

then it should use localhost by default, or you can be more specific via :devtools {:devtools-url ""}

IIRC chrome doesn't allow you to eval anything in extensions anymore though. so even if the websocket connects, you might not be able to do much after

Thanks as always,Thomas! Just got to my computer and did a quick verify with additional :devtools

:target :esm
:devtools {:devtools-url ""
           :use-document-host false}

I'm assuming the watch is running and actually reachable? are you looking at the correct devtools console? IIRC its not in the regular one for the page you are on

Yes, watch is indeed running. I'm not sure if it's at other devtools, but I can see my normal http://localhost/foo requests from my content script, so I'm assuming it's the right place?

the one opened via <chrome://extensions/>. thats the devtools for the actual extension

thanks, i'm googling on how to use it.

just click the Inspect views thing listed there. it opens the devtools in a new window

Got it. But I didn't see any network requests there, not even my local http requests.

I don't know what you are doing to begin with, so I'm just guessing

that's ok, thanks

I have never written a chrome extension with a content script, so I don't even know what that looks like or how the script is loaded

usually with a running watch shadow-cljs should be logging something somewhere when it is loaded. at the very least that the connection failed.

but I don't know the rules for content scripts. maybe websocket is just blocked?

I assume your own code loads fine and works as expected? and the :modules you are using are all correct? as in the devtools are actually loaded in the module that the content script is in?

I use it with emacs cider, it shows below after jacking in or connecting, then after I change some code, it will re-build automatically, the repl output looks like:

;;  Startup: /usr/bin/npx shadow-cljs -d nrepl/nrepl:1.3.0 -d cider/cider-nrepl:0.50.3 -d cider/piggieback:0.5.3 server
;;
;; ClojureScript REPL type: shadow
;; ClojureScript REPL init form: (do (require '[shadow.cljs.devtools.api :as shadow]) (shadow/watch :webapp) (shadow/watch :chrome-ext) (shadow/nrepl-select :webapp))
;;

...
[:chrome-ext] Configuring build.
[:chrome-ext] Compiling ...
[:chrome-ext] Build completed. (510 files, 0 compiled, 0 warnings, 0.77s)

I meant more like the build config for :chrome-ext and how this script is actually loaded

I also wondered whether ws was banned in content script in chrome. but I added some code for manually connecting to the ws server yesterday, I could see some failure messages in chrome devtools (now I don't see this ws message). The code is like below(previously I thought it would just work to have a ws connection from the browser, but I realized it should eval things from shadow cljs, so I am trying to figure out how to do it with shadow-cljs's mechinary):

(when-not (repl/alive?)
    (repl/connect ""
                  :verbose true
                  :print #{:repl :console}
                  :on-error #(print "Error! " %)))

I don't know what that is? what is the repl alias there?

shadow-cljs works by injecting a preload into the build. a preload is loaded before your own sources. thus whatever your code does should happen after shadow-cljs stuff already started

never mind, repl is [weasel.repl :as repl] from a lib named weasel, just for verifying ws capability.

there isn't anything you'd call in your own code ever

weasel is an entirely different thing and not compatible with shadow-cljs

good to know that. I'm not going to use weasel.

you should be able to verify that the preload (just a namespace) is loaded somehow? I mean the requests for your own files should show up somewhere right?

https://github.com/thheller/shadow-cljs/blob/d12811cfb7884d86dcecb3a16cf61c9e89bb8b05/src/main/shadow/cljs/devtools/client/browser.cljs#L1

this is the injected ns

I could totally see chrome blocking that because of eval and stuff, but that should be at least visible somewhere? not just ignored and continue loading your own files?

Not sure if this is the right way, but I do found there is a http request for <chrome-extension://xxxxxxxxx/js/cljs-runtime/shadow.cljs.devtools.client.env.js> (with filter devtools), but it's not the browser ns.

yep, that should be loaded right before the browser one

still haven't seen your build config. I assume you are not setting something like :runtime? i.e. using the default and not changing it?

oh, is this wrong?

:runtime :custom

lol

yes, that is wrong

so it should be :browser ?

that would be the default yes

wow! at least now I see the ws is connected! let me verify if i can do repl stuff. a huge thanks!

I doubt it 😉

https://github.com/thheller/shadow-cljs/issues/902

seems like people went with a custom chrome build to get it to work. no clue how to even do that

okay, you're right. in the repl, I just type in 1 for a quick verify, it complains that No available JS runtime. I think it's should be a runtime connected, right? The code should at least be sent to the browser side.

if the websocket actually is connected there should be a runtime

hmm

http://localhost:9630/runtimes should be listed here

should say "shadow-cljs ready" in the console somewhere

I think I've visited issue 902 before, I'll come back to it later.

well, the runtime is listed there

and you are using that repl?

(shadow/watch :webapp) (shadow/watch :chrome-ext) (shadow/nrepl-select :webapp) suggests otherwise

You can also try the REPL tab in the UI

that should let you select the thing

Cool, just use the REPL UI.

kinda expected that it doesn't work

There is a message in chrome console: browser.cljs:39 Failed to load cljs/user.cljs EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'wasm-unsafe-eval' 'inline-speculation-rules' :* :* ". I'm gonna check out the issue you just sent, and see if there is a recipe.

I might actually have an idea how to get around this eval problem, but assuming that'll also be blocked. just didn't find the time to test this yet. basically the node esm version I wrote also doesn't use eval https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/cljs/devtools/client/node_esm.cljs

well, no I created a version that didn't use eval but didn't end up using it

https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/cljs/devtools/client/node_esm.cljs#L112-L116

its still there though. that might work in chrome, no clue haven't tested 😛

Per issue 902, there is no way to run REPL in a vanilla chrome. A possible workaround is to custom build a chromium for allowing unsafe-eval .

I don't get your node_esm idea, how is it possible to have a repl withing eval'ing, just like you mentioned in gh issue 902.

you could do a quick test to see if dynamic import is allowed

I'm just gonna assume chrome blocks it outright, but who knows

hmm, Idk what dynamic import is, let it google it first

we just need to run code and import() runs code when loading files without using eval

the code is literally just there. you will find nothing to google

in your own file do

(esm/dynamic-import "data:text/javascript;charset=utf-8;console.log(1);")

also need to add (:require [shadow.esm :refer (dynamic-import)]) first though

ok, just me a few minutes and let it test it real quick

regardless. I'd be kinda surprised if chrome allows this but who knows

if it logs 1 somewhere then there is hope 😉

I gotta go, bb in a few hours

unfortunately, it doesn't work, as we expected:joy:

it says: cljs_env.js:1265 Refused to load the script 'data:text/javascript;charset=utf-8;console.log(11111);' because it violates the following Content Security Policy directive: "script-src 'self' 'wasm-unsafe-eval' 'inline-speculation-rules' http://localhost: http://127.0.0.1: <chrome-extension://xxxx-xxx/>". Note that 'script-src-elem' was not explicitly set, so 'script-src' is used as a fallback.

OK, seems the last resort I can think of is to develop it in a web env, let me isolate the extension part first. I was working on this last night, but I was not fully convinced that REPL in an extension doesn't work. Now I am 😞

It's lame, but I think it should work.😂

Thank you, Thomas! Let me know if you think there are docs somewhere that I can update this info to, I would love to help a bit. Thanks for your time and the great project!

thank for confirming that the data: uris are blocked as well, saves me some time testing.

:esm currently does not support a node REPL. only browser.

I'm always amazed by how quick you are to reply, thank you so much for all your hard work. The whole project is about marrying macchiato with the payload CMS v3, which is an ESM-only dependency. Is there any way to require it without switching to :esm? I read something about dynamic requires being an alternative, but couldn't find much about the practice of it.

not really no

Ok, gotcha… I think I'll have to bite the bullet then. Thanks anyway… negative answers still save a lot of time!

why do you ask this in a thread not related to workers at all?

@thheller you where spot on I don't 100% get it but it was not a shadow issue, the ci process was installing the packages with bun which seems to have worked but has created some state that was causing an issue I pinned bun to an older version and everything was working again, what I don't really get is when I cat out package.json it looks fine after the bun command but maybe I am missing something or its due to something happening inside the node_modules folder, anyway pinning to an older version seems to have resolve the issue, thanks for the suggestions.

note that this is not caused by your own projects package.json file, but rather one from a package in node_modules.

Thanks for clarifying that explains why it all looked good from my side

yeah that looks like invalid JSON. not cache related.

if you want a quick debug check to see which file is actually causing the problem try npx shadow-cljs clj-repl

then paste this

(ns shadow.build.npm)

(defonce orig shadow.build.npm/read-package-json)

(defn read-package-json [npm file]
  (try
    (orig npm file)
    (catch Exception e
      (throw (ex-info (str "failed to read package.json file: " (.getAbsolutePath file)) {:file file})))))

and run the build again via (shadow/compile :app)

only case I have seen this before is an aborted npm install that somehow was in the middle of writing a package.json file and basically truncated it mid way

Great thanks for the suggestions @thheller I will give this a try when I am back at my laptop