Clojurians Log v2

Tried (:refer-clojure :only []) with the hope of manually loading squint_core from the right URL - nah, the original import is still there.

you can leave out imports with :elide-imports true

how are you using/generating squint code?

I have squint.edn with

{:paths      ["squint/"]
 :output-dir "squint/"}

Hmm, --elide-imports/`:elide-imports` will work but that would mean that I'd have to provide proper paths inside each .cljs file. Ideally, I imagine that there would be some option in squint.edn that allows inlining an import map, so that all the imports use the real path from the get go.

Ah, wait - :elide-imports removes not just the default import, but all imports. Then I don't know how I'd use it given that I still need squint_core and a bunch of other stuff.

@p-himik what if you do the rewriting yourself based on the import? or what if you post-process the JS with esbuild before you use it in the web worker?

you could bundle the JS with esbuild, I mean

Right now I'm using squint compile with nothing else at all. Adding anything else to the chain would defeat the purpose for me personally - at that point I'd switch to the regular CLJS. Of course, I know that Squint is in a different niche altogether, but a trivial setup was the main point why I wanted to try it and Scittle.

sure. what if you just wrote the import map yourself fully? you can do this for other libs. for squint_core I believe there may be an option, not sure, I'd have to check

I mean, what if you wrote (:require ["" :as foobar])

esbuild isn't a big addition imo btw.

npx esbuild index.mjs --bundle --minify > out.mjs

That's exactly what I've been doing for thirdparty libs already, just because I don't want to go into my HTML for some CLJS change. But the import map is required for Squint itself, as far as I can tell.

ok, so it's just about squint itself. let me check the options

Yeah, by this > that would mean that I'd have to provide proper paths inside each .cljs file. I meant only Squint. In my relatively simple project, all third-party stuff is never imported more than once, so it's pretty much irrelevant.

even if you import it more than once it doesn't matter, browsers cache these imports

so you're not fetching this lib another time from the network - not even loading it a second time

unless you add a parameter to make it unique like

Right, I'm just saying that any thirdparty lib is not a concern at all, but Squint imports themselves completely prevent me from using web workers unless I use additional tools.

understood. checking if there is an existing option and if not, I'm pretty sure we can make this work

so there are several other imports besides the core import that are affected by this:

(case alias
      (squint.string clojure.string) "squint-cljs/src/squint/string.js"
      (squint.set clojure.set) "squint-cljs/src/squint/set.js"
      (if (symbol? alias)
        (if-let [resolve-ns (:resolve-ns env)]
          (or (resolve-ns alias)
              alias)
          alias)
        alias))

"import * as squint_html from 'squint-cljs/src/squint/html.js';\n"

cc/*core-package* "squint-cljs/core.js"

--squint-package=...?

Just to take into account potential needs of others, or maybe even myself down the line - do you think that maybe it might make sense to extend it to import maps in general? BTW found an old closed issue: https://github.com/squint-cljs/squint/issues/224

yes, that issue is closed under the assumption that import maps work :)

And if you think that targeting a wider use case is useful, then no need to think of a name. :) :import-maps.

yeah we can support import maps maybe but maybe it's annoying to have to specify all package like:

"squint-cljs/src/core.js" "https://
"squint-cljs/core.js" "https://
"squint-src/src/html.js" "htps:///

so something like:

--import-maps=<import-maps.json?

which would read that file, and replace accordingly?

> maybe it's annoying to have to specify all package like: Wait, how would that be affected by the hypothetical new feature? Isn't that already what one has to do?

Ah, you probably meant that if the new feature is more focused, then it's just squint-cljs with a single root somewhere, not a line per distinct import.

yeah. but it would probably still be nice to have this feature for other libs

> so something like [...] Yeah! And I'd be fully content if it's also embeddable in squint.edn as proper EDN and not filthy JSON. :D

hehe

I'll just add import-maps support for the squint core pkg now so you can experiment with it. Then we'll extend further support. Does it sound good?

do you use clojure.string etc too?

also import-maps supports "squint-cljs/ as a wildcard prefix, I'll not add support for this yet, just literal replacement so you are not blocked

this seems to work what I have locally:

$ cat lib/other_ns.mjs
import * as squint_core from '';
squint_core.println("macros2/debug", (1) + (2) + (3) + (4));
squint_core.println("macros2/debug", (1) + (2) + (3));

with:

{:paths ["src" "src-other"
         "resources"]
 :output-dir "lib"
 :copy-resources #{"foo\\.json"  "test\\.json" :css}
 :import-maps {"squint-cljs/core.js" ""}}

and no test failures, so I can deploy this for you to test

Sure, sounds perfect. Yes, I do use clojure.string. Thanks! Will try it out.

ok, so support for clojure.string would be nice too then... let me check real quick

Wouldn't that be handled anyway with "squint-cljs/src/squint/string.js" in the import maps?

I didn't handle that part of the code yet. but I did now:

$ cat  /Users/borkdude/dev/squint/test-project/lib/other_ns.mjs
import * as squint_core from '';
import * as str from 'squint-cljs/src/squint/string.js';
squint_core.println("macros2/debug", (1) + (2) + (3) + (4));
squint_core.println("macros2/debug", (1) + (2) + (3));

oh wait no, it didn't work

ah yes:

$ cat  /Users/borkdude/dev/squint/test-project/lib/other_ns.mjs
import * as squint_core from '';
import * as str from '';
squint_core.println("macros2/debug", (1) + (2) + (3) + (4));
squint_core.println("macros2/debug", (1) + (2) + (3));

ok, release coming up... a few minutes

after that we can add better support for import maps with prefixes. this will take more time and less hacks. :)

and will require a Github issue.

ok, try squint-cljs@0.8.154

GH issues + PRs welcome for improvements ;)

:import-maps {"squint-cljs/core.js" ""
                   "squint-cljs/src/squint/string.js" ""}

hmm, crap, looking at JS docs, should this have been :importmaps instead?

or:

{:importmaps {:imports ...}}

Mmm, maybe just :imports then? Since that's the exact feature.

Or [:import-map :imports] - I myself wouldn't use importmap because it's ugly and I can't think of a reason why it should remain ugly and correspond perfectly to what must be going on in HTML.

yes, I think :import-map :imports'. let me change that in the next iteration and I'll ping you about the change. have to do other stuff now.

Sure. Thanks!

import * as squint_core from '';
import * as str from '';

I guess you can just not use importScripts maybe?

or not use a module worker but a classic one

or just pre-bundle the stuff with esbuild anyway. it's not that much of an annoyance (to me at least).

in a module webworker you do:

import * from ... 

The first one is one kind of PITA where I cannot use a non-ESM script without jumping through some hoops. The second one is another kind of PITA where I cannot use Squint to implement that worker. The third one is yet another kind of PITA where I have to host that library on my own server, and it's 11 MB (OpenCV.js). Sigh

ChatGPT is in the chaotic evil mode right now:

const legacyCode = `
  // legacy-lib.js contents
  function legacyAdd(a, b) { return a + b; }
  self.legacyAdd = legacyAdd;
`;

const workerCode = `
  ${legacyCode}
  self.onmessage = e => {
    postMessage(self.legacyAdd(e.data[0], e.data[1]));
  };
`;

const blob = new Blob([workerCode], { type: "application/javascript" });
const worker = new Worker(URL.createObjectURL(blob), { type: "module" });

> Using ImportScripts inside a Module Worker is disallowed Like... why? Who hurt them? As if it's impossible to circumvent. Just throwing wrenches in all the gears for no apparent reason.

you can use a non-ESM script in an ESM script probably using async await + eval? something like (pseudo-code):

const body = fetch(non-esm-scrip).then(x) => x.text())
await eval(body)

Hmm, perhaps.

or vice versa: you can use ESM code in a non-ESM worker with async functions. You can use REPL mode for this squint and then load the script in an async function with dynamic import.

async function() { import("my-esm.js") }

or maybe you don't even need the squint REPL output for this

that's right. you can just import regular ESM code from a non-ESM environment using dynamic import

so that should be a way out

Like a static JS shim that's used for a web worker from Squint code and by itself requires Squint code? Hmm, that's probably the best way to proceed... Thanks again!

I don't know exactly what your setup looks like but the general principle is that async + dynamic import lets you do ESM stuff in non-ESM contexts

something like this (chatGPT):

<html>
  <body>
    <script>
      const worker = new Worker("worker.js"); // classic worker
      worker.onmessage = e => console.log("Main thread got:", e.data);
    </script>
  </body>
</html>

async function init() {
  // dynamically load the ESM file
  const module = await import('./my-esm.js');

  // call something exported
  const msg = module.greet("world");
  postMessage(msg);
}

init();

Seems like that works!

Oh. Oh god. Why haven't I been using (:x obj) and (let [{:keys [x y]} obj] ...) and (assoc! obj :x 1 :y 2) instead of (.-x obj) and other boilerplate. It's so nice.

:)

one more note about import map imports: they are ordered and this is significant. so I think we'll have to use something like a vector of pairs or so

ok now I convinced chatGPT that this order doesn't matter

ah it literally says so in the spec: > • The object properties' ordering is irrelevant: if multiple keys can match the module specifier, the most specific key is used (in other words, a specifier "olive/branch/" would match before "olive/"). >

Told you - ChatGPT is in the chaotic evil mode. :D BTW, the macros are supposed to work, right? Trying to measure some performance because moving OpenCV.js to a web worker made it perform about an order of magnitude worse. Doing this

(defmacro trace [form]
  `(let [start# (.now js/performance)
         result# ~form]
     (log [~(str form) "took" (- (.now js/performance) start#)])
     result#))

var trace = function (_AMPERSAND_form, _AMPERSAND_env, form) {
    return clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(cljs.core.symbol("let")), squint_core.list(squint_core.vec(clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(cljs.core.symbol("start__196__auto__")), squint_core.list(clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(cljs.core.symbol(".now")), squint_core.list(cljs.core.symbol("js/performance")))))), squint_core.list(cljs.core.symbol("result__197__auto__")), squint_core.list(form)))))), squint_core.list(clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(cljs.core.symbol("log")), squint_core.list(squint_core.vec(clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(squint_core.str(form)), squint_core.list("took"), squint_core.list(clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(cljs.core.symbol("-")), squint_core.list(clojure.core.sequence(squint_core.seq(squint_core.concat(squint_core.list(cljs.core.symbol(".now")), squint_core.list(cljs.core.symbol("js/performance")))))), squint_core.list(cljs.core.symbol("start__196__auto__"))))))))))))))), squint_core.list(cljs.core.symbol("result__197__auto__")))));

};

ReferenceError: clojure is not defined

Macros should be imported only with (:require-macros ...) and should reside in a different file. This is a difference and limitation of squint.

isn't the time macro available in squint?

I can't just print the time - it's in a web worker, I have to .postMessage it. Well, AFAIK.

Gotcha about macros - missed that detail completely.

squint doesn't have a "whole project analyzer", it's pretty dumb, so it has to know whether something is a macro or not on a file-by-file basis

unless it's a core macro

(╯°□°）╯︵ ┻━┻ The web worker version is slower because fuck me, that's why. OpenCV.js itself simply works slower there than on the main thread, that's it.

because web workers get less priority or so?

ChatGPT says that it could be one of the reasons. And then lists a whole bunch of other potential reasons.

maybe you have your developer console open and the cache is disabled, and this is why it's slower, because it's requesting stuff? or no, you're actually measuring a synchronous calculation

this maybe makes sense: Workers are about prioritizing. The worker is not really about parallelism, that is more of a side benefit, it’s about concurrency and getting things out of the most valuable thread you have, the UI thread. A web worker isn’t about making something take 2 seconds instead of 4 seconds, it’s about doing that thing with the DOM freezing for 0 seconds.

(from a blog I found)

Yeah, no staff is being fetched. Tried profiling - the profiler somehow lies and reports all times as less than what the difference in performance.now() shows. Yeah, it makes sense that with a web worker I wouldn't see a performance increase - that's not really what I was doing it for anyway. But I definitely was not expecting 3 FPS where previously I was getting smooth 30 FPS (processing a video).

maybe try different browsers?

Of course, no significant differences. It seems that: • Web worker performance is slow only at the start - starts with 80 ms, drops down to tolerable 20 ms. Somehow that happens without any reloads. Just each time after starting the processing • However, drawing on an offscreen canvas that was created with .transferControlToOffscreen is just abysmally slow • I'm sending the data at the framerate of my screen (due to requestAnimationFrame), 120 Hz • The main thread implementation is actually able to keep up with that! So it wasn't meager 30 FPS, it was 120 FPS! And even just the processing time of 20 ms on the web worker, without taking retrieving the data and painting it into account, would not be able to cut it

The problem was a lack of back pressure. But the symptom of "the data is processed as quickly as at the start, but the rendering itself starts lagging behind severely" was very disorienting. Especially given that I did not control rendering at all (`.transferControlToOffscreen`) and all the drawing functions worked immediately. It wasn't relevant for the non-worker impl because it has back pressure built-in. If the computation skips a frame - that's it, the frame is skipped, it's completely forgotten.