a) :hashbang false in the :node-script build config

why not :target :esm and skipping rollup?

heh, see “I will admit right up front that I am not a javascript expert”

I have not looked at javy, so dunno what its limitation for the packaged JS are?

i was having trouble getting through the forest, perhaps because of the trees

https://shadow-cljs.github.io/docs/UsersGuide.html#target-esm

(I did try that, but was having issues)

but its probably my inexperience in the js area

let me try again

the workflow for something like this is totally gonna suck though

WASM can't do hot-reload and stuff

in terms of repl, etc

yeah, agreed

I mean you maybe could develop against node and then just use this as a deployment target

all depends on how capable javy is (which again I haven't looked at)

im basically trying to provide a lambda-function like substrate in the backend, so sandboxing is critical as the code will be untrusted

but yeah, that would be the idea

dev-tooling above the wasm target, and then “release” builds go to wasm

its not ideal, but there are a bunch of nice things the wasm sandbox brings too

i would guess the javy limitations are more to do with quickjs vs a full v8 interpreter

but I am not an expert there, either

so far, from a POC perspective, the stack looks promising in terms of bringing clojure(script) to the mix

I will try your suggestions for the hash-bang and :esm

ty!

release builds will likely just work, dev builds will probably never work, so no watch/compile

the code likely doesn't support any ESM features (as in no export/import)

yeah…i was prototyping with release…didnt even try watch/compile as the rollup wouldve been ugly at best

so you can also use the super simplistic :target :single-file I added recently

ah, im intrigued…what does that do?

https://github.com/thheller/shadow-cljs/issues/1122#issuecomment-1822341864

cool…i will check that out as well…ty

basically just compiles everything and puts it into a single file. completely without anything runtime specific

I would like to retain the ability to bring in npm modules…is that precluded with that target?

that still works

or are you still doing all the shadow-cljs magic to bundle everything in

awesome

yep

ty…i will report back on what I find

> I have not looked at javy, so dunno what its limitation for the packaged JS are? One thing I might not have been clear about: This actually all works (at least for my basic POC cljs code, so far). Which is very cool and a testament to shadow-cljs

I think you were getting at the ability for javy to support hot-reload, etc, but I just wanted to emphasize that at least as a release target flow, everything I described worked, which I am thrilled about

not what I meant no

I mean for example in react-native the hermes runtime doesn't support const, so that has to be transpiled to var

runtimes vary greatly in which JS features they support

Got it. Well, so far so good. But perhaps landmines await

await would be another thing 😉

Like I said, I suspect limitations would more derive from quick js

also a while back people got excited about the graaljs runtime, which works but is also a quite limited runtime. so in the end doing something practical with that becomes annoying

yeah as far as I understand the docs the JS is left as is and just executed via quickjs

Yeah, I don’t know graaljs but I’d guess quickjs is in similar camp

> QuickJS is a small and embeddable Javascript engine. It supports the ES2023 specification including modules, asynchronous generators, proxies and BigInt.

thats already far more than graaljs 😛

In any case, my customers just need to do very simple lambda-function like things to do transformations, so this might be good enough to offer a clojure dialect but not enough for a major app

Awesome, that’s good news

actually https://bellard.org/quickjs/quickjs.html#Standard-library this looks quite capable

maybe even REPL capable

so could likely make a :target :quickjs that you can then optionally turn into wasm

That would be awesome. I suspect interest in this area will grow beyond me, soon

busy with other stuff currently, but I'm intrigued enough to check it out sometime soon

I’ll share my POC once I clean up a few things like the hash-bang and maybe :am

Esm

@thheller so I tried the :esm option, and I get something that compiles but when I run it, nothing happens. At a high level, I understand that the issue is likely that nothing understands where the application entry point is when I am only compiling it as an ESM module. I tried to hack it with an init-fn but still no dice. I am a bit out of my depth on the js side, but perhaps you have an idea

Basically I am doing this

{:builds {:app  {:target           :esm
                 :output-dir       "target"
                 :init-fn           shadow-wasm.core/main
                 :modules {:app {:exports {default shadow-wasm.core/main}}}}}
 :lein true}

(ns shadow-wasm.core)

(defn hello []
  (println "Hello, World"))

(defn main []
  (hello))

that config is incorrect

12:08 $ shadow-cljs release app
shadow-cljs - config: /Users/ghaskins/sandbox/wasm/sci/shadow-cljs.edn
shadow-cljs - running: lein run -m shadow.cljs.devtools.cli --npm release app
[:app] Compiling ...
[:app] Build completed. (45 files, 0 compiled, 0 warnings, 4.67s)
(default ) ✔ ~/sandbox/wasm/sci [master L|⚑ 1]
12:09 $ javy compile target/app.js -o target/app.wasm
(default ) ✔ ~/sandbox/wasm/sci [master L|⚑ 1]
12:09 $ time wasmtime target/app.wasm

real	0m0.458s
user	0m1.833s
sys	0m0.119s

{:builds {:app  {:target           :esm
                 :output-dir       "target"
                 :runtime :custom
                 :modules {:app {:init-fn shadow-wasm.core/main}}}}
 :lein true}

ah!

let me try that

I mean technically it was a correct config, just not doing what you expected probably

makes sense

building now

basically the top level :init-fn had no effect, and you just got afile with a default export

I figured, just didnt know the magic to fix

the :runtime :custom is just there to stop some browser-specific stuff being injects since :browser is the default

boom!

$ time wasmtime target/app.wasm
Hello, World

real	0m0.451s
user	0m1.829s
sys	0m0.111s

this is looking really good

now let me see if I can get deps rolled in

but to your earlier suggestion: in this new world its only shadow :esm and javy, no more rollup or the node preamble trickery

I did some brief quickjs tests myself. the runtime is basic JS but has no support for websockets or fetch

but still pretty neat

there are some built-in modules, since shadow-cljs won't be able to bundle them you probably want to set :js-options {:keep-as-import #{"os" "std"}} in the build config

oh, that will be interesting…Im still new in the wasm world, too, so its not clear what ultimate limitations I may face

ok, i will add that

but then you can (:require ["std" :as std]) (std/getenviron)

dunno if those are available in the wasm variant, but quickjs has them

I assume this is what you mean

{:builds {:app  {:target     :esm
                 :output-dir "target"
                 :runtime    :custom
                 :js-options {:keep-as-import #{"os" "std"}}
                 :modules    {:app {:init-fn shadow-wasm.core/main}}}}
 :lein true}

yep

if you feel like doing some benchmarks I'd be interested to see some numbers comparing to node/browser

if you have pointers as to what you’d like to see benchmarked, I can try to give it a whirl

well whatever your code ends up doing

after wasm AOT this is the basic setup time

$ time wasmtime target/app.wasm
Hello, World

real	0m0.052s
user	0m0.033s
sys	0m0.016s

let me try a nodejs version

nah not interested in startup time

got it

more in like runtime perf difference

right, ok

give me some more time to build this up

then ill try to get some numbers

FWIW, there is anecdotal evidence that fetch is somehow supported on top of quickjs at least when running on wasmedge: https://wasmedge.org/docs/develop/javascript/networking#fetch-client

not sure how they do that, though

(I know even less about wasm than I do about js, at least right now)

yeah I think thats just provided by wasmedge rather than quickjs

just console.log(fetch) in a JS files crashes the process

Yeah, i have some digging to do

I just tried this too:

diff --git a/project.clj b/project.clj
index 088703e..4863aae 100644
--- a/project.clj
+++ b/project.clj
@@ -4,5 +4,6 @@
                  [org.clojure/clojurescript "1.11.132"
                   :exclusions [com.google.javascript/closure-compiler-unshaded
                                org.clojure/google-closure-library]]
-                 [thheller/shadow-cljs "2.27.1"]]
+                 [thheller/shadow-cljs "2.27.1"]
+                 [funcool/httpurr "2.0.0"]]
   :source-paths ["src"])
diff --git a/src/shadow_wasm/core.cljs b/src/shadow_wasm/core.cljs
index f1319a2..8757121 100644
--- a/src/shadow_wasm/core.cljs
+++ b/src/shadow_wasm/core.cljs
@@ -1,7 +1,9 @@
-(ns shadow-wasm.core)
+(ns shadow-wasm.core
+  (:require [httpurr.client.xhr :as http]))

 (defn hello []
-  (println "Hello, World"))
+  (println "Hello, World")
+  (http/get ""))

 (defn main []
   (hello))
\ No newline at end of file

but it pukes with

$ time wasmtime target/app.wasm
Hello, World
Error while running JS: Uncaught ReferenceError: 'XMLHttpRequest' is not defined
    at Wg (function.mjs)
    at <anonymous> (function.mjs:279)
    at <anonymous> (function.mjs:286)
    at fg (function.mjs:97)
    at <anonymous> (function.mjs)
    at <anonymous> (function.mjs)
    at d (function.mjs)
    at e (function.mjs)
    at call (native)
    at h (function.mjs)
    at <anonymous> (function.mjs:290)

Error: failed to run main module `target/app.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x61516 - <unknown>!<wasm function 109>
           1: 0x739d0 - <unknown>!<wasm function 169>
           2: 0xbf6de - <unknown>!<wasm function 1050>
    2: wasm trap: wasm `unreachable` instruction executed

real	0m0.469s
user	0m1.849s
sys	0m0.119s

of course, XHR may be a different beast than fetch

This is something that perhaps you understand: So I added this

diff --git a/src/shadow_wasm/core.cljs b/src/shadow_wasm/core.cljs
index f1319a2..45e4fce 100644
--- a/src/shadow_wasm/core.cljs
+++ b/src/shadow_wasm/core.cljs
@@ -1,7 +1,8 @@
-(ns shadow-wasm.core)
+(ns shadow-wasm.core
+  (:require ["http" :refer (fetch)]))

 (defn hello []
-  (println "Hello, World"))
+  (println "Hello, World: " fetch))

 (defn main []
   (hello))

but I get this

$ shadow-cljs release app
shadow-cljs - config: /Users/ghaskins/sandbox/wasm/sci/shadow-cljs.edn
shadow-cljs - running: lein run -m shadow.cljs.devtools.cli --npm release app
[:app] Compiling ...
The required JS dependency "http" is not available, it was required by "shadow_wasm/core.cljs".

Dependency Trace:
	shadow_wasm/core.cljs

Searched for npm packages in:
	/Users/ghaskins/sandbox/wasm/sci/node_modules
http is part of the node-libs-browser polyfill package to provide node-native package support
for none-node builds. You should install shadow-cljs in your project to provide that dependency.

	npm install --save-dev shadow-cljs

See: 

but I already have shadow in the deps

$ cat package.json
{
  "private": true,
  "devDependencies": {
    "@rollup/plugin-node-resolve": "^15.2.3",
    "rollup-plugin-node-builtins": "^2.1.2",
    "rollup-plugin-node-globals": "^1.4.0",
    "shadow-cljs": "2.27.1"
  }
}

so im not sure whats wrong

(ignore the rollup deps, those were from the earlier experiments

I think part of the story here is that it looks like wasmedge might supply a forked version of quickjs

(which I am not getting by building with javy

http is node built-in package, so not a regular npm package

yeah, i think I get it now

if its also built-in for javy you can use :keep-as-import to tell shadow not to bundle it, and instead load it at runtime

I think the deal is that wasmedge has a modified quickjs which supplies certain things like fetch

oh, thats interesting

let me try that

I should tweak that error message I guess, as it does not apply to :target :esm. only :browser builds

makes sense

this explains how the node apis are available in the wasmedge tooling

https://github.com/second-state/wasmedge-quickjs/blob/main/modules/http.js

so they are implementing certain nodejs built-ins in terms of wasi constructs

which means I probably cant use javy, but I can probably still use the shadow :esm on top

but I might need shadow to basically build as if its :node-script but bundled as if its :esm / rollup

or, maybe the :js-imports trick you mentioned

I don't know why you think rollup does anything useful here?

literally the problem is that shadow-cljs by default for :esm tries to bundle ALL dependencies

if can't find http and fails

thus you configure it NOT to bundle that and instead load it at runtime

rollup cannot bundle it either, so there is no difference

I guess the only difference is that you are using rollup with node-builtins, which basically tells rollup to not bundle node built-in

which is identical to what :keep-as-import is doing, just a bit more manual

an alternate is adding :js-options {:js-provider :import}, that will make shadow-cljs NOT bundle anything

https://shadow-cljs.github.io/docs/UsersGuide.html#_third_party_tool_integration

which you should do if you think about using rollup at all, as shadow partially bundling some stuff and then rollup the rest doesn't make sense

:node-script is very specifically targetting node, hence the name. it does not seems to be the correct fit here.

> I don’t know why you think rollup does anything useful here? Sorry, I just meant “need an ESM module, similar to how shadow :esm or rollup provide”

i think I understand what you are saying about the js-imports though

so ill try that

IIU, I think what I need to do is basically have shadow bundle anything that is not a node built-in, and tell it to skip things that are node builtins, and they will be provided by the wasmedge variant of quckjs

then its the best of both worlds and I wont need rollup, as you indicated

yeah, it is best to not bundle anything the runtime provides. as you'd likely just get a worse variant from npm

Hmm, I thought I was beginning to understand, but still stymied

$ git diff
diff --git a/shadow-cljs.edn b/shadow-cljs.edn
index a50167c..4a8cc31 100644
--- a/shadow-cljs.edn
+++ b/shadow-cljs.edn
@@ -1,6 +1,6 @@
 {:builds {:app  {:target     :esm
                  :output-dir "target"
                  :runtime    :custom
-                 :js-options {:keep-as-import #{"os" "std"}}
+                 :js-options {:keep-as-import #{"os" "std" "http"}}
                  :modules    {:app {:init-fn shadow-wasm.core/main}}}}
  :lein true}
diff --git a/src/shadow_wasm/core.cljs b/src/shadow_wasm/core.cljs
index f1319a2..c9861a7 100644
--- a/src/shadow_wasm/core.cljs
+++ b/src/shadow_wasm/core.cljs
@@ -1,6 +1,8 @@
-(ns shadow-wasm.core)
+(ns shadow-wasm.core
+  (:require ["http" :refer (fetch)]))

 (defn hello []
+  (js/console.log fetch)
   (println "Hello, World"))

 (defn main []

$ wasmedge --dir .:. wasmedge_quickjs.wasm target/app.js
ReferenceError: could not load module filename 'http'

oh well…ill keep digging

so I guess its not available at runtime 😛

heh…apparently. Im basically trying to create the cljs equivalent to https://github.com/second-state/wasmedge-quickjs/blob/main/example_js/wasi_http_fetch.js

can you confirm if (:require ["http" :refer (fetch)])

is the right import statement?

yes

hmm..ok..let me sanity check and actually load their .js example

in the docs it says Install WasmEdge-QuickJS. did you do that? seems to be an optional addon

I believe so, yes

im running it like this

wasmedge --dir .:. wasmedge_quickjs.wasm target/app.js

where the target/app.js is the shadow :esm build

but its probably PEBCAC

ill sanity check with their example: I think I am close

you can just create a .js file with

import { fetch } from 'http';
console.log(fetch)

true, thats a good idea

same error, so theres a different problem in my flow

ah, i think I get whats wrong

its not just that I need the wasmedge-quickjs, but I also need to grant the runtime the modules that it refers to

everything works when I am in the wasmedge-quickjs repo checkout

basically this stuff

https://github.com/second-state/wasmedge-quickjs/tree/main/modules

is getting access via the “--dir .:.”

but bottom line, its working now!

thanks for all the patience…i learned a ton

this is def pretty cool

I tested quickjs with a compile/watch build and they both worked. just had to set :devtools {:enabled false}

thats encouraging

there is no live-reload or REPL but still better than waiting for release build all the time

indeed

I think overall the flow of dev in node and release to wasm wont be too horrible

(if its not possible to solve all the way, I mean)

I mean its likely possible to support a repl/hot-reload with those extra modules

its possible with stock quickjs as well using the std package, just not using websockets

if the barrier was the lack of IO etc, yeah, that should bring a bunch to the table

there is a built-in for http requests in quickjs

urlGet(url, options = undefined)
Download url using the curl command line utility. options is an optional object containing the following optional properties:

oh, thats interesting

seems a bit sketch but could totally be enough

shadow currently only supports websockets, but no reason there couldn't be a http polling alternative

makes sense